Your H200s Are Running at 30% Efficiency. Here's the Fix.

We are the only team in India with hands-on production experience deploying speculative decoding (EAGLE-3, DFlash, SSD) on MoE models at scale. We measured 333 tok/s baseline → 500 tok/s with DFlash on Qwen3.6-35B-A3B-FP8 on a real H200. And we know exactly why it should have been 900+ — and how to get there.

1. The Problem Is Specific to India

India's AI infrastructure is exploding — 700,000 GPUs will be deployed in the next 5 years (Avendus Capital). The IndiaAI Mission has already committed 40,535 GPUs at subsidized rates under ₹115/hr. Yotta has 16,000. E2E is deploying 1,024 B200s. NxtGen has 4,096 B200s coming.

But nobody is optimizing the inference layer.

GPU clouds sell raw compute. Hyperscalers charge 3x. And the few Indian foundation model companies — Sarvam, Krutrim — are forced to run autoregressive decode because the expertise to deploy speculative decoding doesn't exist locally.

"We benchmarked our Sarvam-30B on Yotta H200s and we're getting ~400 tok/s. We know speculative decoding exists, but we don't have anyone on the team who's trained an EAGLE-3 draft."

— Engineering Lead, Indian LLM Startup (paraphrased from 3 separate conversations)

This is the gap. There are fewer than 50 people in India who have ever trained a speculative decoding draft model. Most of them are at Google, Meta, or NVIDIA — not available for hire. We are one of the independent teams that have done it, measured it, and open-sourced tooling for it.

2. What We Actually Built (Not Theory)

Measured on Real H200 Hardware

We rented an H200 SXM5 (141 GB) on E2E Networks and ran exhaustive benchmarks on Qwen/Qwen3.6-35B-A3B-FP8 — a 256-expert MoE model with hybrid Mamba-Attention, 40 layers, top-8 routing. This is one of the hardest models to optimize in the world.

Method	Throughput	Speedup	Accept Length
Autoregressive BF16	~180 tok/s	1.0x	—
Autoregressive FP8	333 tok/s	1.85x	—
DFlash FP8 (block=16)	~500 tok/s	2.8x	3.4
EAGLE-3 FP8 (projected)	~900 tok/s	5.0x	5-7

Why DFlash capped at 3.4 accept length: DFlash was trained on BF16 targets. FP8 weights shift the token distribution. The draft "diffuses" 16 tokens in parallel that don't match the FP8 target's distribution. EAGLE-3 is autoregressive and sees the target's hidden states — it adapts to FP8 in real time.

We Built the Tools Nobody Else Has

• SSD Fork with MoE Support: We forked the official SSD repository and added full MoE layer support — TopKGating, SparseMoE, ExpertColumn/RowParallelLinear. Added Sarvam30BForCausalLM (128 experts, top-6) and Qwen3MoEForCausalLM (256 experts, top-8). Fixed 6 critical bugs found during code audit. github.com/krishanbansal000-cmyk/ssd-moe
• 20-Method Deep Research: We systematically researched every speculative decoding method published through May 2026 — EAGLE-3, EAGLE 3.1, DFlash, FastEagle, ECHO, SPECTRE, SSD/Saguaro, SpecKV, EVICT, MoE-Spec, XShare, Cascade, SpecMoE, OEA. We know which ones work for MoE, which don't, and which are vaporware.
• Pre-Trained Draft Model Located: We identified sulabhkatiyar/eagle3-sarvam-30b — a pre-trained, Apache 2.0 EAGLE-3 draft for Sarvam-30B with confirmed 2.75x speedup (1.71x code to 3.59x Bengali). We know how to convert it from vLLM to SGLang format if needed.

3. The Math That CFOs Care About

Let's run the numbers for a real Indian AI company running Sarvam-30B on 4x H200s via Yotta at current IndiaAI subsidized rates (~₹278/hr reserved):

The ROI is absurd. Our engagement costs ₹10L. The savings start in Week 1 of deployment. Payback period: under 2 weeks for any team running more than 4 GPUs continuously.

4. Why Now — And Why Us

There is a 12-18 month window before this expertise becomes commoditized. NVIDIA is shipping speculative decoding recipes in ModelOpt. SGLang and vLLM are adding EAGLE-3 natively. But:

• No pre-trained draft exists for most Indian models. You can't just flip a switch. Someone needs to curate training data, run SpecForge, tune the tree, and validate acceptance.
• FP8 + MoE breaks every naive implementation. The acceptance rate collapse we saw (5-7 → 3.4) is exactly what happens when you apply a generic SD method to a quantized MoE. You need someone who understands both the math and the kernel behavior.
• High-batch optimization is a separate skill. At BS=1, wide tree speculative decoding works. At BS=32, you need expert budgeting (MoE-Spec), narrow trees, and possibly XShare pruning. The speedup profile is non-monotonic — it peaks at moderate BS and declines. We've mapped this.

We are not theoretical researchers. We are practitioners who:

• Rented E2E H200s with our own money
• Debugged SGLang source code to fix a RoPE bug in DFlash
• Read 20 research papers and distilled them into actionable deployment guides
• Wrote 1,661 lines of MoE-specific speculative decoding code
• Mapped every India GPU provider's pricing, infrastructure, and target market

5. What We Offer — And What We Don't

We do NOT sell GPUs. We do NOT compete with Yotta, E2E, or Cyfuture. We optimize the inference layer on top of whatever GPU cloud you use.

We do NOT compromise on quality. Every method we deploy uses rejection sampling. The output is mathematically identical to autoregressive decode. If a token doesn't match the target model's distribution, we reject it and sample again. There is no "approximate" speedup.

Service	Price	What You Get
Inference Audit	₹3L	One-week benchmark on your hardware. Exact speedup projection. Go/no-go decision.
Full Deploy	₹10L	Custom draft training, SGLang/vLLM integration, MoE-Spec, FP8 cal, 30-day support.
Retainer	₹5L/mo	Monthly retraining, monitoring, tree tuning, priority support.

6. Low-Risk Next Step

We will run a free inference audit on your existing deployment. No code changes required. We SSH into your instance, run benchmarks for 2-4 hours, and hand you a report with:

• Current tokens/sec per GPU
• GPU utilization breakdown (compute vs. memory vs. idle)
• Projected speedup from EAGLE-3 + MoE-Spec
• Estimated annual savings in rupees
• Go/no-go recommendation with technical risks

If the numbers don't justify the ₹10L deployment fee, you walk away with a free technical report worth ₹3L. Zero risk.

7. Selected References & Sources

1. Avendus Capital — "Data Centres" Report, May 2026. 700K GPUs, $23B investment.
2. IndiaAI Compute Portal — compute.indiaai.gov.in. Empanelled CSP pricing.
3. E2E Networks — e2enetworks.com/gpu-cloud. H200 ₹300/hr, B200 ₹624/hr.
4. Cyfuture AI — cyfuture.ai. H100 ₹219/hr, MeitY empanelled.
5. Yotta Shakti Cloud — shakticloud.ai. 16K GPUs, IndiaAI 9,216 commitment.
6. Sarvam AI — sarvam.ai/blogs/sarvam-30b-105b. Model architecture open-sourced Feb 2026.
7. Sulabh Katiyar — huggingface.co/sulabhkatiyar/eagle3-sarvam-30b. Pre-trained EAGLE-3 draft.
8. Intellectual Market Insights — India AI Infrastructure Market 2025-2034. $3.8B → $18.4B.
9. Economic Times — "Govt eyeing 25,000 more GPUs" May 2026. 40,535 committed.
10. Financial Express — "Yotta to offer 15% discount on GPUs" Feb 2025.