IndexCache

This repository provides a patch for SGLang and vLLM that enables IndexCache inference acceleration for models using DeepSeek Sparse Attention (DSA), including DeepSeek-V3.2 and GLM-5.

TL;DR: IndexCache eliminates up to 75% of indexer computations in DSA through cross-layer index reuse — achieving up to 1.82× prefill speedup and 1.48× decode speedup with negligible quality degradation. One if/else branch, zero extra GPU memory.

🔍 Motivation

In DSA, the lightning indexer selects the top-k most relevant tokens at each layer to make attention sparse. While cheap per-FLOP, it runs independently at every layer with O(L²) complexity. At long context lengths, it becomes the dominant bottleneck:

At 200K context, the indexer consumes 81% of prefill time.

💡 Key Insight

We measured pairwise top-k index overlap across all 47 DSA layers and found that adjacent layers share 70–100% of their selected tokens:

Cross-layer top-k overlap heatmap. Most indexer computations are redundant.

⚙️ Method

IndexCache partitions layers into Full (F) layers that retain their indexer and Shared (S) layers that reuse the nearest F layer's cached indices:

We propose two complementary approaches:

Both retain only 1/4 of indexers with negligible quality degradation.

📊 Results

30B DSA Model — Speedup (H100)

9 benchmarks virtually unchanged ✅

GLM-5 (744B) — Production Validation

~1.2× E2E speedup with negligible degradation across 10 benchmarks (long-context + reasoning).

🚀 Quick Start

Step 1: Clone SGLang / vLLM

git clone https://github.com/sgl-project/sglang.git
cd sglang
git checkout b638b25b

This patch is built and tested against commit b638b25b. It may apply cleanly to newer versions, but if you encounter conflicts, use this specific commit.

For vLLM:

git clone https://github.com/vllm-project/vllm.git
cd vllm
git checkout 4508532fb

Step 2: Apply the patch

git apply /path/to/indexcache.patch # /path/to/indexcache_vllm.patch for vllm patch

Step 3: Launch with IndexCache

Configure via --json-model-override-args for SGLang or --hf-overrides for vLLM. Two options (take SGLang for example):

Option A — Uniform interleaving

Every N-th layer keeps its indexer:

python -m sglang.launch_server \
    --model-path zai-org/GLM-5-FP8 \
    --json-model-override-args '{"index_topk_freq": 2}' \
    ...  # your other args (tp, dp, etc.)

index_topk_freq=2 → every 2th layer is Full, rest are Shared (50% indexers removed).

Option B — Custom pattern (from greedy search)

Specify per-layer F/S assignment:

python -m sglang.launch_server \
    --model-path zai-org/GLM-5-FP8 \
    --json-model-override-args '{"index_topk_pattern": "FFSFSSSFSSFFFSSSFFFSFSSSSSSFFSFFSFFSSFFFFFFSFFFFFSFFSSSSSSFSFFFSFSSSFSFFSFFSSS"}' \
    ...  # your other args

Each character maps to one DSA layer: F = Full (runs indexer), S = Shared (reuses cached indices).

Default behavior: When neither parameter is set, all layers run their indexer — identical to standard DSA.

Configuration Reference

Which to use?

• index_topk_freq: 4 — Simple, good default. Best with training-aware models.
• Custom pattern — Optimal for training-free deployment. The example above is the greedy-searched pattern for the GLM-5 model.

✅ Supported Models

Any model using DSA indexer benefits from this patch.

📝 Citation

@article{bai2025indexcache,
  title={IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse},
  author={Bai, Yushi and Dong, Qian and Jiang, Ting and Lv, Xin and Du, Zhengxiao and Zeng, Aohan and Tang, Jie and Li, Juanzi},
  journal={arXiv preprint arXiv:2603.12201},
  year={2025}
}

License

This patch is released under the Apache 2.0 License, consistent with SGLang.