NVIDIA Nemotron 3.5 Bolt E4B 4GB: Fast Conversational Edge Agents, Pure RL Post-Training, and DFlash Parallel Speculative Decoding

(Previous) DeepSeek V4.1-Flash-1017: MIT Weights, DeepInfer Engine, Agent Harness Tuning, and the Full-Stack Search/Fetch API Play | (Next) Claude AI + Higgsfield AI: Native Video Generation, MCP Connector Architecture, and Multi-Model Orchestration

Summary

On October 23, 2026, NVIDIA released NVIDIA Nemotron 3.5 Bolt E4B 4GB, a 4-billion effective-parameter (8B total) open-weight language model post-trained from Google’s Gemma-4-E4B-it specifically for latency-critical, memory-constrained conversational and agentic workloads at the edge. Packaged as a 4-bit Q4_K_M GGUF checkpoint calibrated over 1 million tokens of RL rollouts alongside a full-precision BF16 checkpoint, Bolt E4B runs inside a 3.1 GB to 4.9 GB VRAM budget across consumer laptops (RTX 5070), desktop AI workstations (DGX Spark / RTX Spark), and embedded robotics modules (Jetson Orin Nano 8GB and Jetson AGX Orin 32GB).

Three engineering choices set Nemotron 3.5 Bolt E4B 4GB apart from standard small language models:

1. Pure RL Post-Training Without SFT: In controlled ablations, NVIDIA discovered that running Supervised Fine-Tuning (SFT) on synthetic teacher trajectories boosted raw tool-calling scores faster, but damaged Gemma-4-E4B-it’s natural conversational tone and inflated output length. NVIDIA therefore skipped SFT entirely, post-training Gemma-4-E4B-it purely with asynchronous Reinforcement Learning with Verifiable Rewards (RLVR) and RLHF across seven environments in NeMo-RL and NeMo-Gym using a 2:1 non-reasoning to reasoning sample ratio.
2. Up to 80% Lower Token Verbosity & Spoken-First Tool Use: Using group-relative length penalties in its generative reward model (GenRM) and a cyclical length-penalty schedule, Bolt E4B achieves higher accuracy across 21 benchmarks (63.9% reasoning average and 57.3% non-reasoning average) while generating up to 81.3% fewer output tokens than Qwen3.5-4B (672 vs. 3,604 mean tokens in reasoning mode). It also modifies the chat template so that when a turn includes both speech and tool calls, spoken text is emitted before tool execution, and exposes three request-level knobs: enable_thinking, thinking_token_budget, and conversational_mode.
3. DFlash Single-Pass Parallel Speculative Decoding: Alongside the target model, NVIDIA released a 5-layer, 360M-parameter DFlash speculative drafter (trained on 3.79M conversations via NVIDIA Model Optimizer) that drafts 7 tokens in a single parallel forward pass conditioned on 5 intermediate target hidden layers. On an RTX 5070, Bolt E4B + DFlash reaches 272.6 tokens/second (1.21x faster than Gemma-4-E4B-it + Multi-Token Prediction), and on DGX Spark (BF16 NIM) it accelerates generation by 3.56x (71.6 tok/s vs. 20.1 tok/s).

Hugging Face / Official Model Card Summary

Prior Reading Material

To get the most out of this architectural deep-dive, review these prerequisite articles covering NVIDIA’s small-model research, Physical AI world models, edge decision engines, and low-precision inference:

• NVIDIA Nemotron 3.5 Super VL: 120B MoE Multimodal Architecture, 600+ TPS Agentic Inference, and Enterprise Customization: Our baseline look at NVIDIA’s Nemotron 3.5 family, hybrid Mamba-2 MoE architectures, and NeMo RL post-training adaptation recipes.
• Liquid AI: From Liquid Foundation Models (LFMs) to d1-ultra, Edge Decision Engines, and Non-Autoregressive Fast Paths: Examines how sub-5B edge models and fast decision engines eliminate reasoning-trace latency taxes in on-device agentic loops.
• vLLM v0.32.0: /v1/systemone Structured Decisions, NVFP4 KV Cache Everywhere, and Model Runner V2: Details the NVFP4 quantization ecosystem and high-throughput serving primitives used by modern GPU inference runtimes.
• Physical AI & World Models: NVIDIA Cosmos 3 Architecture Deep-Dive: Covers NVIDIA’s Cosmos 3 visual world foundation models—whose edge variant (Cosmos 3 Edge) supplies real-time camera perception directly to Bolt E4B on the Reachy Mini tabletop robot.
• Vision-Language-Action (VLA) Foundations: From Tokens to Torques in Humanoid Robotics: Explains how high-level language planners interface with low-level robotic motion controllers and animation libraries on embedded hardware.

1. The Edge Agent Dilemma: Why “Chatty” Reasoning Models Fail on Robots and Game PCs

Imagine you are building two real-time interactive applications that must run entirely locally without cloud API bills or network round-trips:

1. A tabletop companion robot running on an 8GB NVIDIA Jetson Orin Nano that watches a room through a camera, listens to a human speak, triggers expressive physical head/antenna animations, and answers back out loud.
2. A dynamic video game NPC—say, a blacksmith merchant—running on a gamer’s 12GB RTX 5070 GPU alongside an Unreal Engine 5 render pipeline that already consumes 8 GB of VRAM.

Figure 1. Reachy Mini running locally on an NVIDIA Jetson Orin Nano 8GB. Cosmos 3 Edge supplies real-time visual understanding of the room to Nemotron 3.5 Bolt E4B 4GB, which invokes scripts from the robot’s animation libraries and generates spoken responses. Source: NVIDIA Research.

If you drop a typical 2026 small reasoning model into either of those systems, three things immediately break the user experience:

• The “Silent Stare” Latency Tax (Verbosity): Modern reasoning models are trained to emit thousands of <think> tokens and long, bulleted Markdown essays before answering a simple question. When a player asks a blacksmith NPC “What kind of armor can I buy?”, waiting 8 seconds while the model burns 3,600 tokens of internal monologue—followed by a 400-word bulleted list—destroys the illusion of a real conversation.
• Silent Tool Execution Before Speech: In standard tool-calling chat templates, when an assistant decides to call a function (such as check_shop_inventory() or play_robot_animation()), it emits the JSON tool call first, waits for the tool response, and only speaks to the user at the end. In a voice or robotics loop, the robot freezes silently while tools execute instead of naturally saying, “Let me check the back room for you…” while triggering the lookup.
• The 4GB Co-Residency Wall: On an 8GB Jetson Orin Nano shared with the OS and vision encoders, or on a 12GB desktop GPU running a AAA game engine, the language model cannot exceed 4 GB to 5 GB of total VRAM—including model weights, KV cache, and speculative decoding heads.

Figure 2. Nemotron 3.5 Bolt E4B 4GB controlling an interactive blacksmith trader NPC in real time, combining low-latency in-character spoken dialogue with deterministic game-state tool calls (sell_to_player, buy_from_player) and character animations. Source: NVIDIA Research.

Think of a standard small reasoning model as a meticulous academic essayist: give it a question, and it locks itself in a room to write a five-page footnote-heavy treatise before speaking a word. NVIDIA Nemotron 3.5 Bolt E4B 4GB is engineered instead like an improv stage actor with an earpiece: it stays in character, speaks concise, natural lines suitable for text-to-speech (conversational_mode), acknowledges the user out loud before reaching for stage props (tools), dials its internal thinking budget up or down per turn (thinking_token_budget), and drafts its words seven tokens at a time via a parallel speculative drafter (DFlash).

flowchart TD
    A["User Voice or Game Event Arrives<br/>(Audio / Camera Frame via Cosmos 3 Edge / Player Prompt)"] --> B["Request-Level Runtime Controls Applied<br/>(enable_thinking • thinking_token_budget • conversational_mode)"]
    B --> C["Nemotron 3.5 Bolt E4B 4GB Target + DFlash Parallel Drafter<br/>(4GB Q4_K_M GGUF + 5-Layer Drafter Proposing 7 Tokens/Pass)"]
    C --> D["Spoken-First Tool Chat Template<br/>(Emits Brief Spoken Acknowledgment BEFORE JSON Tool Calls)"]
    D --> E["Edge Action & Voice Execution<br/>(Robot Animation / NPC Inventory Tool + Low-Latency TTS Output)"]

    style A fill:#0f172a,stroke:#38bdf8,stroke-width:2px,color:#f8fafc
    style B fill:#0f172a,stroke:#a855f7,stroke-width:2px,color:#f8fafc
    style C fill:#052e16,stroke:#22c55e,stroke-width:2px,color:#f8fafc
    style D fill:#0f172a,stroke:#f59e0b,stroke-width:2px,color:#f8fafc
    style E fill:#0f172a,stroke:#10b981,stroke-width:2px,color:#f8fafc

2. Base Architecture, Spoken-First Chat Template, and Runtime Controls

Rather than pre-training a new 4B backbone from scratch, NVIDIA selected Google’s recently released Gemma-4-E4B-it (8B total parameters with 4B effective parameters and a 64K context window) as the starting checkpoint for Bolt E4B. As documented in the 18-page Technical Report, Gemma-4-E4B-it already possessed strong multilingual coverage across 35+ languages, native multimodal vision and audio encoders, and a warm, natural conversational tone—making it an ideal candidate for targeted post-training.

Why Spoken Text Comes Before Tool Calls

In Gemma-4-E4B-it’s default chat template, assistant turns that combine natural-language text with tool invocations serialize the tool call first. NVIDIA modified the chat template in Bolt E4B so that assistant text is placed before tool calls and their results:

• Natural Voice Pacing: When an agent needs to query a database, invoke a home-assistant API, or run a robot motion script, Bolt E4B first outputs a short conversational line (“Sure, let me check the thermostat in the living room…”) that can be streamed immediately to a Text-to-Speech (TTS) engine while the tool call executes in parallel.
• Preserved Multi-Step Chaining: Once the tool result returns, the model seamlessly continues its turn or chains follow-up tool calls without losing context across 64K tokens.

Three Request-Level Latency & Style Controls

Instead of deploying separate checkpoints for fast chat versus complex multi-step planning, Bolt E4B exposes three per-request template controls that let developers tune latency and style on every turn:

1. enable_thinking (True / False): Enabled by default (True). When set to False, the model bypasses internal reasoning traces entirely for ultra-low Time-to-First-Token (TTFT) on straightforward conversational turns.
2. thinking_token_budget (Integer): Caps the maximum number of internal reasoning tokens the model is allowed to spend before producing its final answer. When a developer passes a numeric budget, the chat template injects a system prompt constraint (“Ensure that your thinking process is concise and does not exceed <budget> tokens”), allowing applications to enforce deterministic frame-time or voice-turn SLAs.
3. conversational_mode (True / False): Disabled by default (False) so standard benchmarks receive detailed responses. When set to True for voice assistants, game NPCs, or tabletop robots, the chat template injects a specialized brevity instruction selected via NVIDIA’s internal Brevity Benchmark (which scores responses on essential coverage, conversational tone, and sentence conciseness):

   “Keep responses brief, natural, and easy to understand in conversation. Focus on the most relevant points and avoid unnecessary detail unless asked.”

3. The Post-Training Recipe: Why NVIDIA Dropped SFT for Pure Dual-Mode RL

One of the most revealing engineering findings in the Nemotron 3.5 Bolt E4B Technical Report (Section 3.3) comes from an ablation study comparing a traditional Two-Stage SFT + RL pipeline against Pure RL starting directly from Gemma-4-E4B-it.

Figure 3. Architectural overview of Nemotron 3.5 Bolt E4B 4GB: (Left) Pure asynchronous RLVR + RLHF post-training in NeMo-RL and NeMo-Gym with a 2:1 non-reasoning to reasoning sample ratio and 1M-token rollout imatrix calibration; (Right) DFlash single-pass 7-token parallel speculative decoding versus sequential Multi-Token Prediction (MTP).

The SFT + RL Ablation Discovery

Initially, NVIDIA researchers ran Supervised Fine-Tuning (SFT) on Gemma-4-E4B-it using curated trajectories from the open Nemotron-Post-Training-Datasets before running RLVR. They observed a sharp trade-off:

• What SFT Helped: SFT warmed up the model to the exact tool-calling distribution of the training set, allowing subsequent RLVR to reach a slightly higher tool-calling benchmark ceiling in fewer training steps.
• What SFT Broke: Because the SFT datasets were distilled from multiple heterogeneous teacher models, SFT overwrote Gemma-4-E4B-it’s native stylistic distribution—making the model stiffer, less conversational, and significantly more verbose.

For an edge conversational agent where persona fidelity, natural spoken phrasing, and brevity are paramount, degrading conversational tone was unacceptable. NVIDIA therefore eliminated the SFT stage completely and post-trained Bolt E4B purely with Reinforcement Learning.

Asynchronous RLVR + RLHF Across Seven Environments

Bolt E4B was trained using asynchronous RL in NeMo-RL and NeMo-Gym with a global batch size of 256 prompts, 8 rollouts per prompt (2,048 trajectories per step), a maximum generation length of 4,096 tokens for reasoning rollouts, and 1,024 tokens for non-reasoning rollouts. All training datasets were strictly filtered to remove prompts longer than 16,000 tokens and purged of any benchmark overlap (IFEval, IFBench).

The unified RL stage simultaneously optimizes across seven reward environments:

The 2:1 Non-Reasoning to Reasoning Ratio & Cyclical Verbosity Control

Standard RL post-training causes two classic failure modes in small models: first, non-reasoning accuracy collapses if most training compute goes to long chain-of-thought traces; second, RL models quickly learn to “reward-hack” preference models by generating longer and longer responses. NVIDIA solved both through two training innovations:

1. Strict 2:1 Non-Reasoning to Reasoning Batch Ratio: Every prompt in the RL mixture is presented in both enable_thinking=False and enable_thinking=True formats, weighted at a 2:1 ratio in favor of non-reasoning samples inside every batch. This guarantees that zero-thinking voice and game turns remain crisp and accurate while still teaching the model how to reason when enable_thinking=True.
2. Group-Relative GenRM Length Penalties & Cyclical Tightening: Within the GenRM RLHF environment, NVIDIA added a group-relative length penalty applied to both reasoning tokens and final answer tokens across the 8 rollouts generated for each prompt. Rather than holding the length penalty constant (which can prematurely choke exploration on hard reasoning tasks), researchers cycled between tightening and loosening the length penalty while periodically enabling and disabling overlong rollout filtering.

flowchart TD
    A["Google Gemma-4-E4B-it Base Checkpoint<br/>(No SFT Stage — Preserves Natural Conversational Tone)"] --> B["NeMo-RL + NeMo-Gym Asynchronous Rollout Engine<br/>(Global Batch 256 • 8 Rollouts/Prompt • 2:1 Non-Reasoning to Reasoning)"]
    B --> C["7 Parallel Verifiable & Preference Reward Environments<br/>(Tool-Calling • Schema • IFEval • Instruction Hierarchy • GenRM)"]
    C --> D["Cyclical Length-Penalty & Overlong Filtering Schedule<br/>(Group-Relative Penalty on Both Thinking & Answer Tokens)"]
    D --> E["1M-Token RL Rollout imatrix Calibration<br/>(Produces 4GB Q4_K_M GGUF + BF16 Checkpoints)"]

    style A fill:#0f172a,stroke:#38bdf8,stroke-width:2px,color:#f8fafc
    style B fill:#0f172a,stroke:#a855f7,stroke-width:2px,color:#f8fafc
    style C fill:#0f172a,stroke:#f59e0b,stroke-width:2px,color:#f8fafc
    style D fill:#052e16,stroke:#22c55e,stroke-width:2px,color:#f8fafc
    style E fill:#0f172a,stroke:#10b981,stroke-width:2px,color:#f8fafc

Post-Training Quantization via 1M-Token RL Rollout imatrix Calibration

To package the final BF16 model into a 4GB footprint without sacrificing the newly learned tool-calling and instruction-following behaviors, NVIDIA quantized the model to 4-bit Q4_K_M GGUF using llama.cpp’s importance matrix (imatrix) quantization. Crucially, instead of calibrating the importance matrix on generic Wikipedia text, NVIDIA calibrated imatrix using roughly 1 million tokens of actual RL rollouts generated by the selected checkpoint step—ensuring that attention and MLP weights critical to tool schemas and concise conversational turns retain maximum bit precision.

4. Model Benchmarks & Performance: Higher Accuracy at a Fraction of the Tokens

NVIDIA evaluated Nemotron 3.5 Bolt E4B 4GB (Q4_K_M) across two complementary evaluation suites:

1. A latency-and-accuracy Pareto benchmark on an RTX 5090 across six core agentic, coding, math, and general knowledge benchmarks (TAU2-Bench, SciCode, LIVEBENCH, GPQA, AIME 2025, and MMLU-Pro) with an input sequence length of ~3,000 tokens and output sequence length of 50 tokens.
2. A comprehensive 21-benchmark evaluation suite comparing Bolt E4B (Q4_K_M) head-to-head against Gemma-4-E4B-it (Q4_K_M), Qwen3.5-4B (Q4_K_M), and Nemotron-3-Nano-4B (Q4_K_M) in both Reasoning (enable_thinking=True) and Non-Reasoning (enable_thinking=False) modes.

Figure 4. Smart and Fast: Average benchmark score (TAU2-Bench, SciCode, LIVEBENCH, GPQA, AIME25, MMLU-Pro) versus Time-to-First-Token (TTFT in ms, lower/right is better) on an RTX 5090 (Q4_K_M, ~3K/50 ISL/OSL). Circle size represents VRAM footprint (GB). Cloud reference scores (Gemini 3 Flash, Claude Sonnet 4.6, GPT 5.4 mini) are shown on the left axis. Source: NVIDIA Research.

RTX 5090 Pareto Frontier: Outscoring 9B Models in 4.3 GB of VRAM

As shown in Figure 4 and detailed in Table 9 of the Technical Report, Nemotron 3.5 Bolt E4B 4GB (Q4_K_M) achieves an average score of 70.2 across the six-benchmark suite while consuming just 4.3 GB of VRAM and delivering a 103.5 ms TTFT on an RTX 5090. It outperforms every sub-10B open model tested—including Qwen 3.5 9B (64.1 accuracy, 7.6 GB VRAM)—and trails only 26B–35B class models that require 18 GB to 23 GB of VRAM:

21-Benchmark Suite & The 80% Verbosity Reduction

Across the full 21-benchmark suite (spanning IFBench, IFEval, MulDimIF, FollowBench, LOCOMO, WildBench, Arena-Hard-v2, StyleControl, Instruction Hierarchy, TAU2-Bench Airline/Retail/Telecom, BFCL-v3, BFCL-v4, LiveCodeBench, SciCode, AIME25, MATH-500, GPQA-D, MMLU-Pro, and RULER-64K), Bolt E4B leads all 4B-class peers on 15 of 21 benchmarks in reasoning mode and 14 of 21 benchmarks in non-reasoning mode:

Figure 5. More Accurate, Less Verbose: Average accuracy across 21 benchmarks versus mean output tokens (log scale) in Reasoning mode (hollow circles) and Non-Reasoning mode (solid circles). Bolt E4B delivers +5.8 to +6.1 points higher accuracy than Gemma-4-E4B-it and Qwen3.5-4B while generating up to 81.3% fewer tokens. Source: NVIDIA Research.

Look closely at the token-efficiency numbers underlying Figure 5 (from Section 4 of the Technical Report):

The engineering impact of this verbosity reduction is massive:

• In Reasoning Mode: Bolt E4B generates 59.1% fewer output tokens than Gemma-4-E4B-it (672 vs. 1,642 tokens) and 81.3% fewer output tokens than Qwen3.5-4B (672 vs. 3,604 tokens), while beating both by +5.8 and +6.1 accuracy points.
• In Non-Reasoning Mode: Bolt E4B generates 21.9% fewer tokens than Gemma-4-E4B-it (210 vs. 269) and 48.1% fewer tokens than Qwen3.5-4B (210 vs. 405), while outperforming them by +5.9 and +6.4 accuracy points.
• Category Standouts (Reasoning Mode):
  • Instruction Following: IFBench jumps from 40.95 (Gemma-4-E4B-it) to 60.15 (+19.20 points); IFEval rises from 83.92 to 90.35; MulDimIF rises from 62.25 to 73.65.
  • Instruction Hierarchy (Prompt-Injection Resistance): Overall instruction-hierarchy compliance leaps from 66.16 (Gemma-4-E4B-it) and 59.40 (Qwen3.5-4B) to 82.34 in reasoning mode (and from 54.78 to 79.22 in non-reasoning mode).
  • Conversational Quality & Multi-Turn Memory: Arena-Hard-v2 climbs from 33.20 to 44.54 (+11.34), WildBench from 50.28 to 57.87, and long-horizon conversational memory LOCOMO from 53.39 to 59.17.
  • Agentic Tool Use: TAU2-Bench Telecom jumps from 39.9 (Gemma-4-E4B-it) to 64.8 (+24.9 points), and BFCL-v3 reaches 66.96 vs. Gemma-4-E4B-it’s 60.53.
  • Long Context (RULER-64K): Retains 88.88% accuracy across the full 64K window in reasoning mode (87.65% in non-reasoning mode), slightly ahead of Gemma-4-E4B-it (88.24% / 86.44%).

Preserved Audio, Vision, and 36-Language Capabilities

Even though Bolt E4B’s RL post-training was conducted exclusively on English text trajectories, NVIDIA evaluated how well the model retained Gemma-4-E4B-it’s native multimodal and multilingual weights (Sections 5 and 6 of the Technical Report):

• Speech & Audio Comprehension Improves: On speech benchmarks, text-only RL on conversational and instruction-following tasks actually transferred positively to audio inputs! On Audio MultiChallenge, Bolt E4B improves over Gemma-4-E4B-it from 19.03 to 25.22 (+6.19) in reasoning mode and from 15.71 to 21.02 (+5.31) in non-reasoning mode. On VoiceBench, it improves from 78.20 to 80.30 in reasoning mode (72.42 vs. 69.92 in non-reasoning).
• Vision Benchmarks: Retains solid visual understanding on AI2D (80.57), ChartQA (82.56), DocVQA (87.31), and OCRBench (80.60), with a modest dip on MMMU-Pro Vision (42.02 vs. 47.63 on base Gemma-4-E4B-it) attributable to slight vision-language projector drift during text-only RL.
• Multilingual Translation (WMT24++ Across 36 Languages): Translation quality is fully preserved (xCOMET 94.09 vs. 93.89 +0.20 on English -> X, and 92.39 vs. 92.64 on X -> English), with only a minor -1.22 to -1.52 dip on 12-language MMLU-ProX.

5. DFlash Parallel Speculative Decoding: Drafting 7 Tokens in One Pass

Because Bolt E4B already cuts output token counts by up to 80%, the remaining latency bottleneck in interactive edge deployments is the autoregressive decode speed of the 4B model itself—especially on memory-bandwidth-constrained devices like DGX Spark, RTX Spark, or consumer laptops.

To accelerate token generation without altering the target model’s output distribution, NVIDIA trained and released a companion DFlash speculative decoding draft model (nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB-DFlash for llama.cpp and nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16-DFlash-NVFP4 for vLLM / NIM).

How DFlash Beats Sequential Multi-Token Prediction (MTP)

Traditional speculative drafters (such as EAGLE or standard Multi-Token Prediction heads) generate draft tokens autoregressively one step at a time: to propose 5 draft tokens, the drafter must run 5 sequential forward passes before the target model verifies them.

DFlash eliminates sequential drafting completely:

• 5-Layer, 360M-Parameter Parallel Drafter: The DFlash module is a compact 5-layer Transformer (360M parameters) that reuses the frozen token embeddings and LM head of the target Bolt E4B model.
• Conditioned on 5 Intermediate Target Layers: During the target model’s verification pass, DFlash extracts hidden features from 5 intermediate layers of Bolt E4B and injects them into every DFlash layer via context projections alongside the last verified token embedding.
• 7 Tokens Drafted in a Single Forward Pass: Instead of stepping token-by-token, DFlash takes a block of [token, mask, mask, ...] positions and uses bidirectional attention across the drafted block (paired with a 1,024-token sliding-window KV cache) to predict up to 7 draft tokens in one single forward pass (--spec-type draft-dflash --spec-draft-n-max 7).
• Trained on 3.79M Conversations in ModelOpt: NVIDIA trained the DFlash drafter using NVIDIA Model Optimizer (ModelOpt) on 3.79 million conversations (75% reasoning, 25% non-reasoning) regenerated directly from Bolt E4B at temperature = 1.0.

Engineering Note — Why NVIDIA Chose Plain DFlash Over LiLiCorr: In Section 8.1 of the Technical Report, NVIDIA also evaluated LiLiCorr (which attaches a lightweight likelihood-correlation correction head on top of DFlash to refine draft logits and increases Acceptance Length AL by ~5%). While LiLiCorr shines on large target models (27B–35B) where every avoided target forward pass saves massive compute, Bolt E4B’s 4B decode step is already so fast that the extra compute overhead of the LiLiCorr head outweighed the 5% higher acceptance rate at batch size 1. Thus, plain DFlash delivered the highest real-world tokens/second!

DFlash Speculative Decoding Benchmarks (llama.cpp & NVIDIA NIM)

Across both consumer llama.cpp GGUF setups (RTX 5070 and RTX Spark N1X) and full-precision BF16 / NVFP4 NIM deployments (RTX PRO 6000 and DGX Spark), DFlash delivers substantial wall-clock speedups:

Furthermore, for vLLM and NIM deployments, quantizing the DFlash drafter itself from BF16 (2.06 GB) to NVFP4 (0.58 GB) via GPTQ calibration shrinks drafter weight memory bandwidth by 3.6x per step while matching BF16 acceptance lengths (AL = 3.82 vs. 3.84 on MT-Bench, 4.81 vs. 4.84 on HumanEval, and 5.33 vs. 5.38 on GSM8K).

6. Edge Hardware Telemetry: From RTX 5070 Laptops to Jetson Orin Nano 8GB

How does Nemotron 3.5 Bolt E4B 4GB (Q4_K_M) perform across the actual edge devices where game developers, robotics engineers, and local AI builders deploy models? Table 1 in the Technical Report benchmarks Nemotron-3.5-Bolt-E4B-Q4_K_M.gguf using llama.cpp (CUDA backend, batch size 1) under two realistic context profiles:

• Short-Context Interactive Turn (0 / 3K / 50): 0 cached prefix tokens, 3,000 input tokens, 50 generated output tokens (4K context allocation).
• Long-Context Multi-Turn Session (32K / 2K / 256): 32,000 cached prefix tokens in KV cache, 2,000 new input tokens, 256 generated output tokens (36K context allocation).

Notice the consistency of the memory envelope: even with 32,000 tokens of active KV cache plus a 2,000-token new prompt, Bolt E4B stays between 4.2 GB and 5.0 GB of VRAM, leaving ample headroom on an 8GB Jetson Orin Nano for Cosmos 3 Edge vision processing and audio I/O, or on a 12GB RTX 5070 for real-time 3D game rendering.

7. Running Nemotron 3.5 Bolt E4B 4GB Locally (llama.cpp, DFlash & NIM)

NVIDIA provides zero-day support for Bolt E4B across llama.cpp (with native DFlash speculative decoding support), Hugging Face transformers, vLLM, and the official NVIDIA NIM container (nvcr.io/nim/nvidia/nemotron-3.5-bolt-e4b-4gb).

# 1. Download the 4GB Q4_K_M GGUF checkpoint and companion DFlash BF16 drafter from Hugging Face
pip install -U "huggingface_hub[cli]"
hf download nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB \
  Nemotron-3.5-Bolt-E4B-Q4_K_M.gguf \
  --local-dir /Users/username/models/bolt-e4b

hf download nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB-DFlash \
  Nemotron-3.5-Bolt-E4B-DFlash-BF16.gguf \
  --local-dir /Users/username/models/bolt-e4b

# 2. Launch llama.cpp server with DFlash 7-token single-pass speculative decoding
docker run --gpus all --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 \
  --rm -p 8000:8000 \
  -v /Users/username/models/bolt-e4b:/models \
  ghcr.io/ggml-org/llama.cpp:server-cuda \
  -m /models/Nemotron-3.5-Bolt-E4B-Q4_K_M.gguf \
  -md /models/Nemotron-3.5-Bolt-E4B-DFlash-BF16.gguf \
  --spec-type draft-dflash \
  --spec-draft-n-max 7 \
  --ctx-size 65536 \
  -np 1 \
  --port 8000 \
  --host 0.0.0.0 \
  --jinja \
  --reasoning-format deepseek

# 3. Query Bolt E4B with spoken-first tool calling, thinking_token_budget, and conversational_mode
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")

tools = [
    {
        "type": "function",
        "function": {
            "name": "trigger_reachy_animation",
            "description": "Play an expressive physical head and antenna animation on the Reachy Mini robot.",
            "parameters": {
                "type": "object",
                "properties": {
                    "animation_name": {
                        "type": "string",
                        "enum": ["curious_tilt", "happy_nod", "scan_room", "wave_antenna"],
                    },
                    "speed": {"type": "string", "enum": ["slow", "normal", "fast"]},
                },
                "required": ["animation_name"],
            },
        },
    }
]

response = client.chat.completions.create(
    model="Nemotron-3.5-Bolt-E4B-Q4_K_M.gguf",
    messages=[
        {
            "role": "system",
            "content": "You are Reachy Mini, an expressive tabletop companion robot running on a Jetson Orin Nano.",
        },
        {
            "role": "user",
            "content": "Hey Reachy, how do you like your new office room? Look around and tell me!",
        },
    ],
    tools=tools,
    temperature=1.0,
    top_p=0.95,
    extra_body={
        "top_k": 64,
        "chat_template_kwargs": {
            "enable_thinking": True,
            "thinking_token_budget": 128,
            "conversational_mode": True,
        },
    },
)

msg = response.choices[0].message
# Notice how Bolt E4B emits natural spoken text BEFORE the tool call so TTS starts immediately!
print("Spoken response (streamed to TTS first):", msg.content)
print("Parallel tool calls:", msg.tool_calls)

8. Key Takeaways for Edge AI & Agent Engineers

1. Pure RL Beats SFT + RL When Persona & Brevity Matter: If you are post-training a capable base model (Gemma-4-E4B-it) for interactive voice or character agents, distilling heterogeneous SFT datasets can corrupt the base model’s natural cadence and inflate output length. Directly applying multi-environment RLVR + RLHF with a 2:1 non-reasoning to reasoning mix sharpens tool accuracy (+24.9 points on TAU2-Bench Telecom) and instruction hierarchy (+16.2 points) while preserving natural conversational style.
2. Token Efficiency Is the Real Edge Speed Multiplier: Cutting mean reasoning output length from 3,604 tokens (Qwen3.5-4B) down to 672 tokens (-81.3%) via cyclical group-relative length penalties reduces end-to-end user wait time far more than raw kernel micro-optimizations alone.
3. Parallel Speculative Drafters (DFlash) Outpace Sequential MTP at 4B Scale: By conditioning a 5-layer drafter on 5 intermediate target layers and predicting 7 draft tokens in a single bidirectional forward pass, DFlash delivers 272.6 tok/s on an RTX 5070 (+21% over MTP) and up to 3.56x faster decoding on DGX Spark.
4. Spoken-First Chat Templates Are Essential for Physical AI: Reordering the assistant serialization format so spoken acknowledgments precede JSON tool calls eliminates the awkward silence that plagues voice assistants, game NPCs, and companion robots during tool execution.

(Previous) DeepSeek V4.1-Flash-1017: MIT Weights, DeepInfer Engine, Agent Harness Tuning, and the Full-Stack Search/Fetch API Play | (Next) Claude AI + Higgsfield AI: Native Video Generation, MCP Connector Architecture, and Multi-Model Orchestration