vllm.cpp

A 1:1 port of vLLM to pure C++ —
no Python, no PyTorch, no ggml at runtime. It mirrors the V1 architecture and
targets the same algorithms and serving surface; the tables below distinguish
the implemented subset from the remaining parity work. It is usable as a
library (llama.cpp-style) with a C API, an example CLI, and an
OpenAI-compatible server.

⚠️ Pre-release, under heavy development. Both NVFP4 gate models —
Qwen3.6-35B-A3B (MoE hybrid) and Qwen3.6-27B (dense W4A4) — now run the
full paged engine end-to-end on NVIDIA GB10 (DGX Spark, sm_121a) with
token-exact greedy correctness gates passing. Performance parity is
currently GATING. The exact pushed-a531e05 cache-off, closed-loop 27B
online checkpoint validates all 12 performance groups but remains below the
every-axis floor: median total-throughput ratios at c1/2/4/8/16/32 are
0.968/0.934/0.948/0.955/1.003/0.927×, with only
4/2/5/3/10/8 of 20 throughput+latency axes and 2/4 memory axes
passing. The c32 total-throughput outlier includes a confirmed HTTP worker
capacity failure in two of three repetitions; the residual c1-c16 decode gap
includes a confirmed FP4 autotune-key defect. A fresh server started directly
at c16 removes that defect's TPOT gap (161.72-161.75 ms ours versus 161.70 ms
vLLM in the standard series), so HTTP capacity and exact FlashInfer FP4
bucket/tactic parity are the priority repairs. The 35B campaign is held until
27B passes every axis. Historical temperature/token-budget-mismatched ratios
remain diagnostics only. The tables track real, tested support and are kept
current as work lands (see
.agents/). CPU multithreaded dispatch is now
correctness-gated, but its real-file throughput/RSS checkpoint is still open.
The current serving checkpoint implements native-ID final/continuous stream
usage for completion and chat and is CPU/ASan/UBSan/TSan-green; the fresh
two-model online latency rerun remains GATING. The prior 8289cbd
partial 27B arm is void and contributes no public ratio.

What's implemented (CPU, behaviorally tested)

The vLLM V1 engine path needed by the two Qwen gate-model text workloads
runs end-to-end on CPU:

• Engine core — the unified token-budget Scheduler (chunked prefill, FCFS
  preemption), hash-based prefix-caching KVCacheManager + hybrid KV coordinator
  (full-attn + GDN/mamba-state groups), the persistent InputBatch + step-input
  build, EngineCore.step() loop, and the LLMEngine (add_request/step/
  generate).
• Local-attention KV + sliding-window operator — SlidingWindowSpec and
  ChunkedLocalAttentionSpec now have registry-backed managers, bounded
  admission, local prefix reuse/recycling, and hybrid-disabled full-allocation
  fallbacks, all CPU/property/sanitizer-gated. The generic sliding window also
  propagates through config, the CPU reference, separately specialized
  portable-CUDA kernels, and vendored FA2 local dispatch; the sm_121a
  translation units compile cleanly. Chunked-local attention now has its cached
  ordinary-backend wrapper, exact virtual Q/K batches, reusable block-table
  gather/update plan, cudagraph rejection and generic spec seam; all six pinned
  shapes plus randomized causal-mask properties are CPU/sanitizer-gated.
  Neither mode is yet user-visible model support: GPU runtime, positive-model
  e2e, oracle, trace, and performance/memory gates remain open.
• Long-context RoPE foundation (YaRN/MRoPE/Llama 3/Phi-3/dynamic-NTK) —
  modern and legacy Hugging Face RoPE parameters now normalize into a typed,
  memoized factory with f32/bf16 caches.
  Plain YaRN and its mrope_section branch apply caller-supplied caches for
  NeoX/GPT-J layouts and 1-D or 3-axis contiguous/interleaved positions; Llama 3
  adds its exact unchanged/smoothed/scaled frequency bands, including equal
  low/high factors. Phi-3 LongRoPE builds its short and long factor-array caches
  and selects one globally from runtime max length, including explicit mscale
  overrides. Dynamic-NTK mirrors both its factor/trained-length and alpha base
  transforms with pinned alpha-first dispatch. Fifteen exact pinned-source
  oracle fixtures, CPU references, and ASan/UBSan pass. The CUDA implementation
  is source-complete but has not yet been compiled or run on an uncontended
  GPU, and no feature-positive model path is claimed yet.
• Model forward — Qwen3.6-35B-A3B hybrid (GDN×3 + gated full-attention, 256-
  expert MoE + shared expert), with paged attention (block-paged KV cache) +
  batched GDN. Loads from safetensors (NVFP4/FP8→bf16) and GGUF
  (k-quant→bf16 for the qwen35moe APEX files). Model selection now uses a
  central ordered, type-erased registry over the full Hugging Face
  architectures list for both implemented Qwen IDs; unsupported,
  previously-supported, and out-of-tree IDs are rejected explicitly instead of
  being guessed from num_experts. The registry's CPU suite is green; its
  unchanged two-model GPU path is awaiting the queued no-regression rerun.
• CPU execution — persistent ggml-derived native threadpool with chunked
  GEMM and row/batch op dispatch; VLLM_CPP_CPU_THREADS selects the worker
  count. Outputs are byte-identical at 1/3/20 threads and the concurrency suite
  is ThreadSanitizer-clean. The real-file ≥10x speed/RSS checkpoint still needs
  an exclusive idle-host rerun, and GGUF compute-in-quant remains open.
• Sampler — greedy (bit-exact vs torch.argmax) plus temperature,
  top-k/top-p, penalties, min-p and internal logit-filter/logprob primitives.
  n, full random/logprob payload behavior, and request/API wiring for the
  long-tail controls remain open; some paths synchronize results to the host.
• OpenAI server — basic /v1/completions and /v1/chat/completions
  transport (non-streaming plus live incremental SSE over an AsyncLLM
  engine thread), concurrent requests through one scheduler, and disconnect
  abort. Streaming completion and chat requests accept stream_options with
  native-ID final/continuous usage frames; --enable-force-include-usage
  mirrors vLLM's server-wide force mode. This path is CPU/sanitizer-gated and
  awaits its fresh two-model GB10 online checkpoint before the row can close.
  --max-num-seqs and --max-num-batched-tokens expose reproducible scheduler
  operating points. /v1/models, /health, and /version are
  present, while /health
  currently reports process liveness rather than probing engine health. The
  async path passes the full CPU suite and ThreadSanitizer; its post-change
  GB10 token/latency/throughput/memory gates are still pending. Chat templates
  use a bounded minja-subset Jinja engine. --[no-]enable-prefix-caching
  exposes the tri-state cache policy; hybrid/attention-free generation models
  default off like pinned vLLM, while ordinary decoder models default on.
  Tool/function calling provides Hermes-style <tool_call> parsing,
  streaming tool-call deltas, and tool_choice auto/required/named — auto
  is relaxed: the model may reply in plain text or call a tool, constrained only
  once it starts a call; ⚠️ the Qwen3.6 gate checkpoints' shipped chat template
  uses the Qwen3-Coder XML tool format and forced <think> blocks, which the
  current parser (including the local qwen3 Hermes alias) and template engine
  do not yet handle — the
  qwen3_coder XML parser + reasoning parser + template-engine extensions are
  scoped in .agents/specs/mm-tools-scoping-2026-07-10.md); grammars / structured output (JSON-schema,
  json_object, regex, choice, GBNF — a native constrained-decoding engine with
  an explicitly bounded JSON-schema/structural-tag subset behind vLLM's
  structured-output seam).
• Library packaging — a stable 17-symbol C ABI (include/vllm.h),
  libvllm shared + static, an example CLI and OpenAI server, and a
  dlopen/FFI consumption path (for LocalAI-style embedding via cgo/purego).
  Alongside the blocking completion calls, additive request handles provide
  nonblocking submit/cancel/wait/done/error/free with per-delta callbacks.

Every unit is adversarially reviewed and behaviorally tested; the CPU test suite
is green (ctest). See the parity ledger for the
per-change record vs upstream vLLM.

Quick start

# Build (CPU; add -DVLLM_CPP_CUDA=ON on an NVIDIA box, -DVLLM_CPP_SERVER=ON for the server)
cmake -S . -B build -DVLLM_CPP_SERVER=ON
cmake --build build -j
ctest --test-dir build            # the behavioral suite

# Fast-GDN CUDA build (the MVP-parity numbers): adds the Triton-AOT GDN kernels.
# They are VENDORED as pre-compiled cubins (src/vt/cuda/triton_aot_vendored/) —
# building needs ONLY a C compiler, no Python/Triton (regenerating the vendored
# artifacts is a maintainer task: scripts/regen-triton-aot.sh).
cmake -S . -B build -DVLLM_CPP_CUDA=ON -DVLLM_CPP_TRITON=ON

# Serve a supported Qwen text checkpoint (safetensors dir or supported 35B .gguf)
./build/examples/server --model /path/to/Qwen3.6-35B-A3B --port 8000 \
  --max-num-seqs 32 --max-num-batched-tokens 8192
# then: curl localhost:8000/v1/chat/completions -d '{"model":"...","messages":[...]}'
# add --enable-force-include-usage to attach native cumulative usage to every
# streamed choice and an empty-choice final usage frame before [DONE].

# Or a one-shot completion via the CLI (drives the C ABI end-to-end)
./build/examples/vllm-cli --model /path/to/model --prompt "The capital of France is"

As a library, link libvllm (or dlopen it) and drive the C API in
include/vllm.h: use vllm_complete / vllm_complete_stream for blocking
calls, or vllm_request_submit plus the request lifecycle functions for
nonblocking concurrent streams.

Supported model architectures

Acceleration

Kernels & target hardware

The ops exercised by the two GB10 text gates have portable references and CUDA
implementations grounded in vLLM or its dependencies. This is not a claim of
complete vLLM-kernel or cross-architecture coverage: the canonical kernel and
CUDA-target inventories track unimplemented and untraced families separately.

Only GB10/sm_121a is built, traced and gated today. Source-level fallbacks
suggest routes for other SMs, but none counts as support until its full build,
correctness, trace and performance block passes. Non-CUDA backends
(Metal/Vulkan/ROCm/XPU) are post-MVP and will port from the CPU reference.

Quantization

Legend: ✅ supported & tested · 🚧 in development · 🗓 planned.

Status & caveats (honest)

• Where we are (GB10 / DGX Spark, sm_121a): both NVFP4 gate-model text paths
  run the full paged engine end-to-end, and their real-model token-exact greedy
  correctness gates pass. The CUDA kernel paths exercised by those gates are
  validated on real hardware.
• Production-vLLM performance is below the exact gate. The historical 35B 1.0195× and
  27B 1.007× offline ratios were produced with vLLM's
  bench throughput default temperature=1.0, while vllm.cpp used greedy
  temperature=0; the 27B vLLM arm also resolved an 8192-token scheduler budget
  while vllm.cpp used 2048. Those measurements and their memory observations
  remain useful diagnostics, but they are not identical-workload acceptance
  evidence. The replacement pushed-a531e05 27B online series is exact and
  binding: all 12 groups validate, but only c16 median total throughput exceeds
  vLLM and no concurrency passes all 20 performance axes. Host PSS/RSS also
  remain above vLLM. Two c32 repetitions stranded one accepted HTTP connection
  for roughly 205-207 seconds; direct c16 trace starts separately prove that
  the current FP4 key aliases M=1/2/4/8/16 and reuses the first M=1 tactic.
  These are concrete repair targets, not a hardware ceiling. The exact 35B run
  waits behind 27B closure.
• The optimized paths themselves remain implemented: vendored Triton-AOT GDN,
  cuBLASLt TN projection layouts, fused attention preamble, tiled causal-conv,
  vendored FlashAttention-2 prefill, register-tiled GDN delta_h, tensor-core WY,
  fused fp8 RMSNorm→quant, bf16 GDN I/O, and default-on fp4 autotuning. The
  current FP4 implementation is not yet FlashInfer-equivalent: its small-M cache
  bucket aliases five distinct runtime shapes and exposes only four wide
  persistent tactics, while the traced vLLM dependency selects from a wider
  static-persistent/Stream-K family. Until repaired, isolated per-shape wins do
  not establish end-to-end tactic parity.
  Historical same-binary component A/Bs remain evidence for those individual
  levers; they do not substitute for the reopened end-to-end oracle gate. The
  full record is in the parity ledger, and
  scripts/dgx-bringup.sh runs the CUDA suite and gates on dgx.casa.
• Device-resident cache work is active. W0 gives CUDA full-attention KV and
  GDN convolution/recurrent state stable vt::Alloc ownership and
  stream-ordered zeroing. W1 adds persistent device metadata plus one indexed
  BF16↔F32 gather/scatter launch per mixed GDN layer, including fresh-state
  zeroing and mixed decode/prefill turnover semantics; CPU retains host storage.
  VT_DEVICE_KV_CACHE=0 restores the former CUDA host-vector path for an exact
  W0 A/B, while VT_GDN_INDEXED_STATE_IO=0 restores W1's row-copy baseline.
  The current local serial CPU suite passes 105/105, as do focused ASan/UBSan
  access suites and a clean CUDA 13.0.88/sm_121a build. A strict local model
  LeakSanitizer run still
  reports 58,624 bytes in 153 process-lifetime scratch-pool allocations; the
  indexed op itself is leak-clean.
  Both 35B/27B tests pass in default indexed mode and in fallback mode; the W0
  default tests assert every cache pointer is CUDA device memory. On
  the exact cache-off c16/48 workload, three interleaved repetitions average
  781.80 indexed vs 776.95 fallback tok/s (+0.6246%) for W1, with all 20
  throughput+latency axes better, 0.846%/0.530% CV, and all six server
  lifecycles returning memory. Its traces collapse cudaMemcpyAsync calls from
  163,540 to 7,508, D2D calls from 142,717 to 1,231, and D2D traffic from
  49,088 to 1,856 MB. The profiler perturbs the two arms unequally and
  inverts their throughput, so only the unprofiled AB/BA/AB series supplies the
  speed ratio; the trace supplies structural evidence. Separate W0 27B/35B
  compute-sanitizer runs pass 234/234 and 315/315 assertions with zero access
  errors, and the W1 indexed-op memcheck passes 7/7. Full leak checking remains
  red on pre-existing bounded
  caches (47.29 MB dense; 36.82 GB MoE/Marlin); W0's multi-GiB cache allocations
  are absent from both leak inventories and all server lifecycle returns pass.
  This component win does not close the full-grid vLLM throughput/host-memory
  gate.
• CPU status: the native threadpool and chunked op dispatch pass the complete
  suite at 1/3/20 workers and a focused ThreadSanitizer run. This is not yet a
  speed claim: the required same-binary 1-vs-20 Qwen3.5-2B GGUF throughput/RSS
  series was deferred because the recovered 20-core host was contended by
  unowned inference services. It remains GATING until an exclusive idle-host
  run reproduces the ≥10x prefill/decode floor and refreshes llama.cpp.
• No PyTorch / no ggml at build or runtime (ggml is a format reference for GGUF
  only). Original runtime/packaging components (the vt op runtime, the minja
  chat-template engine, the native grammar backend, the C API) are documented as
  deviations in the porting inventory §9.
• The CUDA low-concurrency comparison against SGLang is specified and pinned to
  v0.5.13. Its CPU-only P1 corpus/client/summary/process-memory/dry-run harness
  is implemented and contract-tested, but no image, model compatibility, or
  SGLang performance result is claimed yet. The unchanged 27B checkpoint has
  a plausible native path; exact 35B mixed-quant loading must be proven, and
  pinned-client raw E2E/TPOT detail remains a loud preflight gap. True
  incremental async HTTP streaming exists, but binding TTFT/ITL numbers wait
  for the post-W2 GB10 online gate (SERVE-ASYNC-LLM remains GATING, not
  DONE). Historical 31d053f and offline ratios are non-binding because the
  audit found cache, sampling, token-budget, admission, and model-length
  mismatches. Pushed f065fce completed the first exact 27B cache-off,
  closed-loop baseline: 2,016/2,016 timed requests, three repetitions, six
  memory returns, the real-model gate, and both corrected traces pass their
  contracts. Mean total-throughput ratios are below the required floor at every
  point: 0.962/0.926/0.936/0.945/0.978/0.972× from c1 to c32; only 3–5/20
  timing axes and 2/4 memory axes pass. Dispersion is low (throughput CV <0.7%),
  so this is the binding 27B gap baseline, not noise or a parity pass. The
  corrected nsys trace records 153,394 cudaMemcpyAsync calls; persistent GDN
  state rows alone account for 20.809 GiB H2D plus 20.806 GiB D2H across its
  three repetitions. That grounds device-resident KV/GDN state as the first
  repair. W0 is merged at 7d29e0c; its clean GB10 build, default/fallback
  35B+27B pointer/token gates, lifecycle, trace, and repeated 27B A/B pass. The
  measured W0 same-binary gain is +2.12% (20/20 timing axes). W1 then wins a
  separate same-binary A/B by +0.6246% (20/20) and collapses the traced copy
  loop from 163,540 to 7,508 async calls. Neither component result is a fresh
  vLLM denominator or may be multiplied into the old grid. Both model paths are
  W0 memory-access clean and the W1 indexed op is memcheck-clean, but the
  required zero-leak result remains open on inherited process-lifetime pools.
  Fresh exact direct-library/online 27B and 35B every-axis closure remains
  mandatory before later roadmap work; W2 direct indexed convolution-state
  update is the next scoped lever if the fresh 27B run remains below floor.
• Speculative decoding is not user-visible yet. The first MTP leaf now has
  safetensors loaders and a standalone dense/MoE Qwen3.5 head with CPU tests,
  but its exact 27B+35B oracle gate is still queued and the scheduler,
  rejection sampler, GDN state snapshots, and API/config wiring remain open.
  GGUF files do not currently carry mtp.*; normal non-speculative execution
  is unchanged.
• Local attention is not user-visible yet. Sliding-window KV bookkeeping
  and backend-neutral CPU/portable-CUDA/FA2 compute-window leaves are
  implemented; chunked-local KV sizing, registry/grouping, fixed-chunk prefix
  reuse/recycling, admission and fallback allocation are also implemented, as
  is the virtual-batch wrapper that delegates each fixed chunk to an ordinary
  causal backend. Their CPU/property/sanitizer gates pass, and the
  sliding-window sm_121a translation units compile. GPU runtime, supported
  StarCoder2/Gemma3/Llama4-class model paths, oracle, trace, throughput, latency
  and memory evidence remain open; all four execution rows remain GATING, not
  supported.
• Scaled long-context RoPE is not user-visible yet. Typed config, memoized
  f32/bf16 cache construction, plain/MRoPE YaRN, Llama 3 frequency-band, Phi-3
  LongRoPE short/long, and both dynamic-NTK formulas, plus the CPU supplied-
  cache operator, pass fifteen exact pinned-source oracle fixtures and
  sanitizer checks. CUDA compile/runtime, Nomic/Qwen-VL YaRN, Llama-3.1,
  Phi-3, and Hunyuan consumers, both-engine traces, and correctness/performance/
  latency/memory closure remain open; all four scaled-RoPE execution rows are
  GATING, not supported.

Project record

Development is fully documented under .agents/ (canonical index in
AGENTS.md): the gates, the porting inventory
vs upstream vLLM, the parity ledger, the roadmap,
the engine, model,
quantization, kernel,
and backend matrices,
and the upstream sync protocol. The concise public
accepted/pending/failed/void benchmark checkpoint is in
docs/BENCHMARKS.md.

License

Licensed under the Apache License, Version 2.0. See NOTICE
for third-party attributions (this project is a C++ port of the Apache-2.0-licensed
vLLM and vendors additional components under
their respective licenses).