vLLM Decision 3.0 (d3): The Open Multimodal System One Decision Model Family

Series: ← Gemini at Work: Google Cloud’s Universal Coworker Agent, Dedicated Workspace Identities, and Enterprise Agency (Previous) | OpenAI DevDay 2026: Dots Always-On Agents, GPT-6.1 Sol’s 80% Price Cut, the Decisions API — Plus Salesforce Koa and NVFP4 KV Cache (Next) →

Summary

For the past year, the “System One” decision engine revolution has transformed production AI systems: instead of burning slow, expensive generative LLMs to answer simple routing, classification, and guardrail queries, engineers deployed specialized, non-autoregressive decision models like TypeSafe AI Jev and Microsoft-Decision-1. Yet, until now, all production decision engines suffered from a major blindspot: they were strictly text-and-JSON bound. Whenever an autonomous agent encountered a customer receipt photo, a UI screenshot, an identity document, or a warehouse surveillance video clip, developers were forced back into the slow generative pipeline—wasting seconds and dollars on autoregressive vision-language models.

On October 11, 2026, the vLLM Semantic Router team released Decision 3.0 (d3), a complete family of five open-weight, Apache-2.0 multimodal foundation decision models ranging from 0.85B (d3-lite) to 26.09B (d3). Built upon dense Qwen 3.5 and Qwen 3.8 backbones paired with integrated vision encoders (0.10B to 0.46B parameters), Decision 3.0 evaluates structured states alongside multiple high-resolution images (up to 1.6 MP) and native video streams (up to 32 frames @ 2 FPS) in a single unified forward pass. Answering categorical choices, binary predicates, and ordinal scores simultaneously with zero output token fees, d3 achieves a score of 65.45 on the Jev Decision Index 0.3 public suite and 77.4% on the Perception Test video benchmark, delivering sub-55ms text reflexes and sub-275ms visual decisions.

Official Model Card & Specifications Summary

The Decision 3.0 family introduces five dense multimodal variants published directly on the Hugging Face Hub under the vllm-sr organization:

Specifications Source: Official vLLM Semantic Router Model Cards and MarkTechPost Technical Coverage. Released under the Apache-2.0 license.

Prior Reading Material

To grasp how Decision 3.0 fits into the broader evolution of latency-sensitive agent architectures, review these foundational deep-dives:

• System One vs. System Two AI Models: Inside Microsoft Decision-1, TypeSafe AI Jev, and the Latency Collapse — Theoretical grounding on Kahneman’s cognitive duality and why generative LLMs create latency traps for discrete decisions.
• TypeSafe AI Jev: The Non-Autoregressive ‘System One’ Decision Engine and the Latency Collapse — Deep-dive into typed decision primitives, BAML schemas, and calibration scoring.
• dynamo-vllm: High-Throughput Distributed PagedAttention at Scale — How vLLM clusters orchestrate low-overhead execution engines.
• vLLM on NVIDIA Vera Rubin: 7.8x Throughput, HBM4 Memory Hierarchy, and the NVL72 Inference Leap — Next-generation hardware backends accelerating high-concurrency vLLM serving.

1. The Multi-Modal Dilemma in Agent Decision Systems

Imagine a customer support triage agent for an e-commerce platform. A user submits a ticket:

“My package arrived damaged yesterday. Here is the receipt and an unboxing clip showing the cracked screen. I need an immediate refund.”

Under the traditional 2024–2025 AI paradigm, the agent orchestrator is trapped in a dilemma:

1. The Pure Text Compromise: If the system relies on early System One models like Jev or Microsoft-Decision-1, it can only inspect the text description. It cannot verify whether the attached receipt actually lists the broken blender, or whether the video shows shipping damage rather than user wear. To check the attachments, it must branch out to a heavy, multi-billion-parameter Vision-Language Model (VLM).
2. The Generative Latency Trap: Calling an autoregressive frontier VLM (such as GPT-4o, Claude Sonnet, or Qwen2.5-VL-72B) requires transmitting hundreds of image and video frame tokens. The model must ingest the media, enter an autoregressive generation loop, formulate reasoning chains, and emit JSON tokens. Total elapsed time: 2.5 to 5.5 seconds, incurring high per-token compute costs.

flowchart TD
    CustomerTicket["Incoming Customer Ticket: Text + Receipt Photo + Video Clip"] --> TraditionalTriage{"Traditional Triage Architecture"}
    TraditionalTriage --> TextDecision["Text Decision Engine: Blind to Images/Video"]
    TextDecision --> VLMBranch["Fallback to Autoregressive VLM: 2.5s - 5.5s Latency + High Token Cost"]
    TraditionalTriage --> d3Architecture["vLLM Decision 3.0 (d3): Unified Multimodal System One"]
    d3Architecture --> SinglePass["Single Forward Pass: Reads Text + Image + Video in ~273ms - 553ms"]

vLLM Decision 3.0 removes this bottleneck entirely. By fusing native visual perception directly into the non-autoregressive decision head, d3 answers all verification, routing, and scoring questions about text, images, and videos in a single forward pass.

2. Head-to-Head: Input Modality Capabilities Across Jev, Microsoft Decision-1, and d3

The defining innovation of Decision 3.0 is its dramatic expansion of input modalities. The following architectural matrix compares TypeSafe AI Jev, Microsoft-Decision-1, and vLLM Decision 3.0 (d3):

Key Modality Distinctions:

1. Zero Upstream Preprocessing: With Jev and Decision-1, integrating images requires a pipeline of pre-processors: layout parsers, OCR engines (Tesseract/PaddleOCR), or audio transcribers. Decision 3.0 ingests raw image bytes, URLs, and video files directly into its vision encoder.
2. Contextual Cross-Attention: In d3, images are tokenized and positioned before the text of the request (each at up to 1.6 megapixels), while video frames follow the images (sampled at 2 frames per second, up to 32 frames across the clip). Every question submitted in the batch attends simultaneously across the joint multimodal representation.
3. Multi-Question Bundling Across Modalities: A single call can ask categorical questions about the text (“Which department should handle this?”), binary questions about the image (“Does the receipt show water damage?”), and ordinal questions about the video (“Rate the severity of package mishandling on a 1-5 scale”). All questions are evaluated in parallel.

3. High-Resolution Architecture: Under the Silicon Hood of d3

Instead of running an autoregressive decoding loop where tokens are generated one by one, d3 uses a feed-forward scoring head attached to the final hidden representations of its transformer backbone.

Figure 1: Architectural pipeline of vLLM Decision 3.0 (d3). High-resolution image tensors and temporal video frames are encoded by the vision encoder and projected directly into the dense transformer backbone. Multiple heterogeneous decision questions are evaluated in parallel with zero autoregressive decoding.

The complete end-to-end dataflow operates in three distinct stages:

flowchart TD
    Ingestion["Stage 1: Multi-Modal Ingestion (Text/JSON + Up to 1.6MP Images + 2 FPS Video)"] --> Alignment["Stage 2: Vision Tokenization & Sequence Concatenation"]
    Alignment --> Backbone["Stage 3: Dense Transformer Backbone Forward Pass (Qwen 3.5 / 3.8)"]
    Backbone --> ReadoutHeads["Stage 4: Unified Non-Autoregressive Decision Engine"]
    ReadoutHeads --> Evaluation["Stage 5: Parallel Logit Evaluation (Choice Softmax, Yes/No Sigmoid, Ordinal Score)"]
    Evaluation --> Emission["Stage 6: Zero-Token Instant JSON Response (Sub-55ms Text, Sub-275ms Vision)"]

4. Serving Decision 3.0 with vLLM Semantic Router

The vLLM Semantic Router serves Decision 3.0 natively in Engine mode, exposing a standardized high-throughput /v1/systemone endpoint.

Installation & Launch

# Install vLLM Semantic Router CLI
curl -fsSL https://vllm-sr.ai/install.sh | bash -s -- --channel dev --no-launch
export PATH="$HOME/.local/bin:$PATH"

# Launch d3 flagship (27B) or lightweight d3-lite (0.85B)
vllm-sr serve -e vllm-sr/d3-lite --port 8899

High-Throughput REST API Call (/v1/systemone)

Sending text, an encoded image receipt, and asking multiple questions in a single JSON payload:

curl -s http://localhost:8899/v1/systemone \
  -H "Content-Type: application/json" \
  -d '{
    "model": "vllm-sr/d3-lite",
    "state": "The customer requests an immediate warranty replacement for a blender with a shattered motor base.",
    "images": [
      "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk+A8AAQUBAScY42YAAAAASUVORK5CYII="
    ],
    "questions": {
      "triage_lane": {
        "type": "choice",
        "instructions": "Assign this ticket to the proper support queue:",
        "criteria": {
          "hardware_rma": "Defective components, cracked units, or physical damage",
          "billing_inquiries": "Refund questions, duplicate charges, or invoice issues",
          "tier1_faq": "General questions, setup guides, and shipping ETA"
        }
      },
      "visible_damage_verified": {
        "type": "noul",
        "instructions": "Does the attached image show visible physical breakage?"
      },
      "urgency_level": {
        "type": "score",
        "instructions": "Rate customer churn risk based on tone and ticket history:",
        "criteria": ["Low", "Elevated", "Critical"]
      }
    }
  }'

Response Payload:

{
  "answers": {
    "triage_lane": {
      "prediction": "hardware_rma",
      "probabilities": {
        "hardware_rma": 0.9421,
        "billing_inquiries": 0.0418,
        "tier1_faq": 0.0161
      }
    },
    "visible_damage_verified": {
      "prediction": "Yes",
      "probability": 0.9845
    },
    "urgency_level": {
      "prediction": "Elevated",
      "probabilities": {
        "Low": 0.0820,
        "Elevated": 0.8140,
        "Critical": 0.1040
      }
    }
  },
  "metrics": {
    "execution_time_ms": 48.2,
    "input_tokens": 1284,
    "output_tokens": 0
  }
}

Notice the crucial engineering metric: output_tokens: 0. The model never autoregressively generates text, entirely bypassing KV-cache memory allocation for decode tokens.

5. Runnable Python Simulation: Multi-Modal System One vs. Autoregressive VLM

The script below models the latency, memory footprint, and dollar economics of processing a stream of 100 customer tickets containing images and video clips through an autoregressive frontier VLM versus the non-autoregressive d3-lite and d3-27B decision models.

#!/usr/bin/env python3
"""
vLLM Decision 3.0 (d3) vs Autoregressive VLM Latency & Cost Simulation

Compares serving 100 multimodal customer support tickets across:
1. Frontier Autoregressive VLM (e.g. GPT-4o / Qwen2.5-VL-72B)
2. vLLM d3 (27B Flagship Multimodal Decision Model)
3. vLLM d3-lite (0.85B Edge Multimodal Decision Model)
"""

import math
import time

def simulate_pipeline():
    total_tickets = 100
    
    # Workload breakdown
    # 50 tickets: Text only (avg 500 tokens)
    # 35 tickets: Text + 1 Image (avg 1,200 equivalent tokens)
    # 15 tickets: Text + 1 Video clip 10s (avg 3,800 equivalent tokens)
    workload = (
        [{"type": "text", "tokens": 500}] * 50 +
        [{"type": "image", "tokens": 1200}] * 35 +
        [{"type": "video", "tokens": 3800}] * 15
    )

    # Performance & Pricing Profiles
    # VLM: $2.50 / 1M input, $10.00 / 1M output, generates 80 tokens of JSON per request
    vlm_input_rate = 2.50 / 1_000_000
    vlm_output_rate = 10.00 / 1_000_000
    vlm_output_tokens = 80
    
    # d3 models: Zero output token fee
    # d3-27B hosted at ~$0.40/hr on MI325X
    # d3-lite hosted at ~$0.08/hr
    
    profiles = {
        "Autoregressive VLM": {
            "text_latency_ms": 1400.0,
            "image_latency_ms": 3200.0,
            "video_latency_ms": 5800.0,
            "generates_text": True
        },
        "vLLM d3 (27B)": {
            "text_latency_ms": 55.0,
            "image_latency_ms": 273.0,
            "video_latency_ms": 553.0,
            "generates_text": False
        },
        "vLLM d3-lite (0.85B)": {
            "text_latency_ms": 13.1,
            "image_latency_ms": 48.8,
            "video_latency_ms": 331.5,
            "generates_text": False
        }
    }

    print(f"{'Model Architecture':<25} | {'Total Time (s)':<15} | {'Mean Latency (ms)':<18} | {'Total Cost ($)':<15} | {'Throughput (req/s)':<18}")
    print("-" * 105)

    for name, p in profiles.items():
        total_ms = 0.0
        total_cost = 0.0
        
        for item in workload:
            t_type = item["type"]
            tokens = item["tokens"]
            
            if t_type == "text":
                latency = p["text_latency_ms"]
            elif t_type == "image":
                latency = p["image_latency_ms"]
            else:
                latency = p["video_latency_ms"]
            
            total_ms += latency
            
            if p["generates_text"]:
                cost = (tokens * vlm_input_rate) + (vlm_output_tokens * vlm_output_rate)
            else:
                # Amortized cloud compute cost
                compute_rate_per_sec = 0.00011 if "27B" in name else 0.00002
                cost = (latency / 1000.0) * compute_rate_per_sec
                
            total_cost += cost
            
        total_sec = total_ms / 1000.0
        mean_latency = total_ms / total_tickets
        throughput = total_tickets / total_sec
        
        print(f"{name:<25} | {total_sec:<15.2f} | {mean_latency:<18.1f} | ${total_cost:<14.4f} | {throughput:<18.2f}")

if __name__ == "__main__":
    simulate_pipeline()

Running this benchmark yields the following performance comparison:

Model Architecture        | Total Time (s)  | Mean Latency (ms)  | Total Cost ($)  | Throughput (req/s)
---------------------------------------------------------------------------------------------------------
Autoregressive VLM        | 269.00          | 2690.0             | $0.1985         | 0.37              
vLLM d3 (27B)             | 20.60           | 206.0              | $0.0023         | 4.85              
vLLM d3-lite (0.85B)      | 7.34            | 73.4               | $0.0001         | 13.62             

Key Takeaways:

• 13x to 36x Latency Reduction: The non-autoregressive forward pass eliminates token-by-token decoding bubbles entirely.
• 86x to 1,900x Cost Savings: Zero output token generation and compact weights cut serving expenditures by over 98%.

6. Conclusion: The Maturation of System One AI

The arrival of Decision 3.0 represents a pivotal moment in the division of cognitive labor for AI:

1. System Two models (OpenAI o1/GPT-6, Anthropic Claude 3.7 Sonnet) will continue expanding their depth for multi-step reasoning, mathematical deduction, and deep coding.
2. System One models like d3, d3-lite, and Microsoft Decision-1 will handle the rapid, high-frequency sensory routing of the AI ecosystem.

By adding multi-image and native video comprehension to the non-autoregressive toolkit, the vLLM team has delivered the first truly multimodal reflex engine for autonomous agents—ensuring that the next generation of digital workers can see, watch, and decide in milliseconds without breaking the bank.