Back to the trend of the day: China's DeepSeek gives away the engine that runs its AI, free
Poly Rating · from the trend of the day

vLLM vs SGLang vs llama.cpp: Top 9 engines, Oct 2026

On the same 8× H200 server running DeepSeek-R1, one published run puts vLLM ahead (1,137 vs 1,041 tokens/s) and the next puts SGLang ahead (1,329 vs 1,042), so 'which is faster' depends on how you test.

  • No. 1: SGLang
  • 9 places

01Final list

The leading AIs each built their own list of ten independently. Here is one result that is easy to check: every place has a reason, a source, and the list of AIs that included it.

Order: from most to least AI agreement.

A bright badge means this AI put the entry on its list. A dim badge means it did not.

  1. 1

    SGLang

    Keeps Huawei Ascend, AMD, TPU and Apple support in the main project. Beats or trails vLLM on DeepSeek-R1 depending on prompt count (3,276 vs 2,499 tok/s at 200 prompts).

    Included it: Claude, ChatGPT, Grok, Gemini, Kimi, Wenxin. Did not include: Meta.

    In their lists: Claude: 2 · ChatGPT: 2 · Grok: 1 · Gemini: 2 · Kimi: 2 · Wenxin: 2

    What each AI noted 6
    • Claudeplace 2

      It treats Ascend as part of the shared model code, with _is_npu branches in the main files, and AMD, TPU and Gaudi routes on top.

      Source ↗
    • ChatGPTplace 2

      At 200 prompts, SGLang leads vLLM by about 31% on the cited 1k/2k H200 workload; with fewer prompts, it loses.

      Source ↗
    • Grokplace 1

      One README names AMD, TPU, Intel, Apple and Huawei next to Nvidia — and one H100 test has it at 16,215 tokens/s against vLLM's 12,553.

      Source ↗
    • Geminiplace 2

      Crushes agentic and multi-turn latency via RadixAttention, outpacing standard runtimes by up to 3x on shared prefix workloads while offering native ROCm support.

      Source ↗
    • Kimiplace 2

      RadixAttention makes it the agent-loop specialist, and its Ascend port officially supports more frontier open models - DeepSeek V4.1, GLM-5.3, Kimi K3, MiniMax-M3, Hy4 - than any other engine.

      Source ↗
    • Wenxinplace 2

      Faster than vLLM in head-to-head on the same GPUs — and beat it to Ascend support

      Source ↗
    6 AIs put it here
    Source ↗
  2. 2

    llama.cpp

    Widest single code tree: CUDA, HIP, Metal, Vulkan, SYCL, CANN (Ascend), MUSA. Ran the full DeepSeek-R1 on 8 AMD MI300X at 37 tok/s decode, per AMD.

    Included it: Claude, ChatGPT, Grok, Gemini, Kimi, Wenxin. Did not include: Meta.

    In their lists: Claude: 3 · ChatGPT: 3 · Grok: 3 · Gemini: 3 · Kimi: 3 · Wenxin: 4

    What each AI noted 6
    • Claudeplace 3

      The widest chip list of any engine here: Metal, CUDA, HIP, Vulkan, SYCL, CANN (Ascend) and MUSA in one tree, but built for single-user speed rather than datacenter batches.

    • ChatGPTplace 3

      One codebase reaches Apple Metal, AMD, Intel SYCL, Vulkan and other backends, making it the broadest practical hardware escape hatch here.

      Source ↗
    • Grokplace 3

      CANN, HIP, Metal, Vulkan and SYCL sit in one tree — the same tree that, on one A100 write-up, serves a 70B model at roughly a third of vLLM's batch rate.

    • Geminiplace 3

      The ultimate portable escape hatch from enterprise GPU monopolies, serving 70B models seamlessly across Apple Metal, CPU AVX-512, and ROCm with zero CUDA dependencies.

      Source ↗
    • Kimiplace 3

      One C++ codebase carrying CUDA, ROCm, Vulkan, Metal, SYCL and MUSA backends - it runs the full 671B DeepSeek-R1 on 8 AMD MI300X GPUs at ~37 output tok/s with no vendor fork.

      Source ↗
    • Wenxinplace 4

      Runs 70B models on a single MacBook — the people's engine, even if it's not the fastest

      Source ↗
    6 AIs put it here
    Source ↗
  3. 3

    DeepInfer (DeepSeek)

    Runs on Nvidia SM100 and Huawei Ascend 950, but supports one model, offline only, with three commits at launch. Its 25,000 tok/s per GPU is DeepSeek's own figure.

    Included it: Claude, ChatGPT, Grok, Gemini, Kimi, Wenxin. Did not include: Meta.

    In their lists: Claude: 5 · ChatGPT: 4 · Grok: 4 · Gemini: 4 · Kimi: 5 · Wenxin: 3

    What each AI noted 6
    • Claudeplace 5

      Released one day ago and already described as running about 155 trillion tokens a day, but it supports one model on two chips.

    • ChatGPTplace 4

      It pairs an unusually important Ascend 950 target with a launch-day codebase of only three public commits and no published Ascend throughput.

      Source ↗
    • Grokplace 4

      Day-old, 14.1k stars, two chip families in one repo — and it runs one model, with the cache left as homework.

    • Geminiplace 4

      DeepSeek's battle-tested 155-trillion-token-a-day engine, built with custom FlashMLA and DeepGEMM kernels targeting Nvidia SM100 and Huawei Ascend 950.

      Source ↗
    • Kimiplace 5

      One day old and already the only engine co-designed with a frontier model: DeepSeek's own production stack, tuned for V4.1-Flash on Nvidia SM100 and Huawei Ascend 950 from release.

      Source ↗
    • Wenxinplace 3

      Born free on Oct 20 2026 — runs DeepSeek's own 155T-token workload on Nvidia and Ascend alike

      Source ↗
    6 AIs put it here
    Source ↗
  4. More about the top three ↓
  5. 4

    TensorRT-LLM

    The Nvidia speed yardstick: 87,444 tok/s under MLPerf's latency cap vs vLLM's 71,588. Supports no non-Nvidia chips.

    Included it: Claude, ChatGPT, Grok, Gemini, Kimi, Wenxin. Did not include: Meta.

    In their lists: Claude: 6 · ChatGPT: 5 · Grok: 5 · Gemini: 5 · Kimi: 4 · Wenxin: 5

    What each AI noted 6
    • Claudeplace 6

      The speed reference at about 2,400 tok/s on H100 in one comparison, with zero non-Nvidia chips.

    • ChatGPTplace 5

      It remains the Nvidia-focused speed reference, but its value as a route away from Nvidia is effectively zero.

      Source ↗
    • Grokplace 5

      The Nvidia yardstick: 87,444 tokens/s under MLPerf's latency cap, and zero non-Nvidia chip families.

      Source ↗
    • Geminiplace 5

      Nvidia's proprietary speed crown: unmatched raw throughput on H100 and Blackwell, but completely locked to the Nvidia hardware ecosystem.

      Source ↗
    • Kimiplace 4

      The speed king on Nvidia - 2,780 tok/s on Llama-3.3-70B at 100 concurrent requests vs vLLM's 2,400 - paid for with a 28-minute engine compile and zero non-Nvidia support.

      Source ↗
    • Wenxinplace 5

      Nvidia's own speed king — fastest on H100/B200 but useless if you want off-Nvidia

      Source ↗
    6 AIs put it here
    Source ↗
  6. 5

    Ollama

    Estimate: Easiest local start (~184k stars) and inherits llama.cpp's chip list. One cited test has it at ~41 tok/s under load against ~793 for vLLM.

    Included it: Claude, ChatGPT, Grok, Gemini, Kimi, Wenxin. Did not include: Meta.

    In their lists: Claude: 7 · ChatGPT: 9 · Grok: 6 · Gemini: 8 · Kimi: 7 · Wenxin: 6

    What each AI noted 6
    • Claudeplace 7

      Easiest to install, but one report has it flatlining near 41 tok/s under load while vLLM peaked near 793.

      Source ↗
    • ChatGPTplace 9

      Its simple local distribution reaches many users, but user convenience and backend portability do not substitute for a controlled serving benchmark.

      Source ↗
    • Grokplace 6

      184,000 stars and a one-line install — it is llama.cpp in a friendlier coat, not a second answer to a full GPU cluster.

      Source ↗
    • Geminiplace 8

      Packaging llama.cpp and custom runners into a single CLI command, turning local non-Nvidia workstations into plug-and-play AI servers.

      Source ↗
    • Kimiplace 7

      The developer on-ramp: one binary wrapping llama.cpp across CUDA, ROCm and Metal - and since 0.19 a preview MLX backend that is 1.6-2x faster at decode on Apple Silicon.

      Source ↗
    • Wenxinplace 6

      Not a raw engine — a polished wrapper on llama.cpp that made local AI mainstream for non-engineers

      Source ↗
    6 AIs put it here
    Source ↗
  7. 6

    Hugging Face TGI

    Hugging Face's server is now in maintenance mode, and its README points users to vLLM, SGLang, llama.cpp and MLX.

    Included it: Claude, ChatGPT, Grok, Gemini, Kimi, Wenxin. Did not include: Meta.

    In their lists: Claude: 9 · ChatGPT: 7 · Grok: 8 · Gemini: 7 · Kimi: 10 · Wenxin: 7

    What each AI noted 6
    • Claudeplace 9

      A production server with automatic quantization and LoRA, but I found no non-Nvidia or DeepSeek-R1 data.

    • ChatGPTplace 7

      A mature, documented deployment project, though the cited measurements do not establish a matching cross-chip speed comparison.

      Source ↗
    • Grokplace 8

      Still the third name on a 70B bake-off: about 2,100–2,600 tokens/s, behind vLLM's 2,800–3,400 and ahead of llama.cpp.

      Source ↗
    • Geminiplace 7

      Hugging Face's hardened enterprise server, providing solid multi-accelerator deployment across Intel Gaudi, AMD ROCm, and Nvidia GPUs.

      Source ↗
    • Kimiplace 10

      Now in maintenance mode, Hugging Face's own Rust server tells users to move to vLLM or SGLang - the formal sign that the serving stack consolidated around two Apache-2.0 leaders.

      Source ↗
    • Wenxinplace 7

      HuggingFace's production-grade wrapper — widely used on cloud but frozen on chip diversity

      Source ↗
    6 AIs put it here
    Source ↗
  8. 7

    LMDeploy

    Its TurboMind core beat vLLM on one H100 test (1,980 vs 1,850 tok/s). Ascend runs only through its PyTorch engine, not the fast C++ core.

    Included it: ChatGPT, Gemini, Kimi. Did not include: Claude, Meta, Grok, Wenxin.

    In their lists: ChatGPT: 6 · Gemini: 6 · Kimi: 8

    What each AI noted 3
    • ChatGPTplace 6

      A substantial serving option with documented non-Nvidia ambitions, but the evidence gathered here does not put it on equal benchmark footing with vLLM and SGLang.

      Source ↗
    • Geminiplace 6

      Shanghai AI Lab's lightweight powerhouse, achieving lightning-fast TurboMind C++ token generation and official Huawei Ascend acceleration.

      Source ↗
    • Kimiplace 8

      The C++ engine that beats vLLM on its own turf - 1,980 vs 1,850 tok/s on a 7B H100 - while its PyTorch engine adds official Huawei Ascend support the TurboMind core lacks.

      Source ↗
    3 AIs put it here
    Source ↗
  9. 8

    MLC-LLM

    Compiles models for Metal, Vulkan, WebGPU and mobile chips without CUDA. No same-model speed test against the server engines.

    Included it: Claude, ChatGPT, Gemini. Did not include: Meta, Grok, Kimi, Wenxin.

    In their lists: Claude: 8 · ChatGPT: 8 · Gemini: 10

    What each AI noted 3
    • Claudeplace 8

      Aimed at edge, mobile and WebGPU, with about 21.6K GitHub stars.

      Source ↗
    • ChatGPTplace 8

      Its compiler-oriented portability broadens device options, but this evidence set does not provide a same-model, same-hardware throughput comparison.

      Source ↗
    • Geminiplace 10

      Compiles deep learning models into standalone executable machine code via Apache TVM, running on Qualcomm, Apple Metal, and Vulkan devices.

      Source ↗
    3 AIs put it here
    Source ↗
  10. 9

    xLLM (JD)

    JD's Apache-2.0 engine, built for Chinese accelerators, scored 11,352 vs MindIE's 8,476 tok/s on DeepSeek-R1 on Ascend. These are the authors' own figures.

    Included it: Claude. Did not include: ChatGPT, Meta, Grok, Gemini, Kimi, Wenxin.

    In their lists: Claude: 4

    What each AI noted 1
    • Claudeplace 4

      Built for Chinese accelerators and claimed at 1.7 times MindIE and 2.2 times vLLM-Ascend on Qwen under equal TPOT, which is the authors' own claim.

    1 AIs put it here
    Source ↗

02More about the top three

The ranking is above. Here is more about the three places at the top: what they are, why they are here, and what is worth knowing.

The three figures are not like for like: SGLang's is on H200 at 200 prompts, llama.cpp's is decode speed on 8 AMD MI300X, and DeepInfer's is DeepSeek's own per-GPU figure.

1SGLang

No. 1 — SGLang. On the lists of 6 of 7 AIs; median place there: 2. It keeps the most non-Nvidia chips (Ascend, AMD, TPU, Apple) inside the main project, and it can beat vLLM on DeepSeek-R1.

Did you know

  • SGLang began as university research at Stanford and UC Berkeley. It now lives under LMSYS, the non-profit that also runs Chatbot Arena. Its trick, RadixAttention, stores past work in a tree so repeated prompt starts are not recomputed.

    Source ↗
What it is
A fast open engine for serving AI models, started as university research.
Its edge
Most non-Nvidia chip guides in the main project, plus the lead over vLLM with many prompts.
Who it suits
Suits teams serving many users on mixed chips; short tests may mislead because of warm-up.
  • DeepSeek-R1 speed (tokens/s)3,276 tok/s at 200 prompts

Wikipedia · Wikidata

Why it is here

  • AMD’s SGLang 0.4.2 test used DeepSeek-R1 FP8 on 8 MI300X GPUs, with concurrency up to 16 and 3,200 input tokens. That gives a stated hardware and workload for judging its throughput. — (voice only)

    Source ↗

  • The main SGLang hardware guide lists 8 platform guides, including AMD, Ascend, TPU, XPU and Moore Threads. That breadth helps explain its first place for non-Nvidia support. — (voice only)

    Source ↗
Did you know

  • The project says SGLang makes trillions of tokens a day across more than 400,000 GPUs, and lists xAI, LinkedIn, Cursor, AMD and NVIDIA as users. The page labels these numbers project-reported, not independently checked.

    Source ↗
History

  • In 2023, SGLang’s paper introduced RadixAttention, a way to reuse cached work when prompts share parts. That gave the project a clear performance idea at its launch: the paper reported up to 6.4x higher throughput in its tests. — (voice only)

    Source ↗
Keep in mind

  • The same forum thread says SGLang needs a warm-up period that vLLM does not. Early requests may be slower than later ones, so a short test can mislead. Try both engines on your own workload.

    Source ↗
Where to go further

  • Open the AMD GPU page and read the install section first. It covers the Docker image for AMD Instinct GPUs such as the MI300X, and it has a DeepSeek-V3 launch example with 8-way tensor parallelism.

    Source ↗
Compare the top three ↓

2llama.cpp

No. 2 — llama.cpp. On the lists of 6 of 7 AIs; median place there: 3. One code tree runs on the widest range of chips, even the full DeepSeek-R1 on AMD, but it is built for single-user speed.

Did you know

  • llama.cpp's first README said it was 'hacked in an evening', and its author had no idea whether it worked correctly. Its main goal was to run LLaMA with 4-bit compression on a MacBook, with Apple chips treated as first-class.

    Source ↗ ✓ confirmed by several AIs
What it is
A small C++ engine that runs AI models on ordinary computers and many chip types.
Its edge
Widest chip list in one codebase: Apple, AMD, Intel, Vulkan, Ascend and more.
Who it suits
Suits one user or a small setup; not people serving many users at once.
  • DeepSeek-R1 speed (tokens/s)37 tok/s on 8 AMD MI300X

Wikipedia · Wikidata

Why it is here

  • AMD reports llama.cpp on 8 MI300X GPUs delivered up to 76% higher throughput than 8 H100 GPUs on DeepSeek-V3-671B, with a 4,096-token prompt. The comparison is a concrete multi-GPU result, though it is not a DeepSeek-R1 test. — (voice only)

    Source ↗

  • On the same 8-MI300X setup, AMD reports up to 213% higher throughput than H100 on Llama 3.1 70B with Flash Attention and a 4,096-token prompt. This gives a second stated model and prompt condition for its throughput case. — (voice only)

    Source ↗
Did you know

  • Within days of the March 2023 release, Simon Willison ran Facebook's LLaMA 7B on a 64GB M2 MacBook Pro. After Georgi Gerganov's conversion script, the model was only about 4GB.

    Source ↗
Keep in mind

  • Even a fan of llama.cpp writes that it "doesn't scale like vLLM." It suits one user or a small setup better than serving many people at once.

    Source ↗
What people say

  • It even felt faster, probably thanks to quantisation and the simplicity of the codepath. — Antonio Poggi

    Source ↗
Where to go further

  • Open the build guide and jump to the HIP (AMD) and CANN (Ascend) sections. You can build several backends at once and choose the device when you run it.

    Source ↗
Compare the top three ↓

3DeepInfer (DeepSeek)

No. 3 — DeepInfer (DeepSeek). On the lists of 6 of 7 AIs; median place there: 4. It runs on Nvidia and Huawei Ascend, but it supports one model, offline only, and is a day old.

Did you know

  • DeepInfer was not written from scratch. Its README says it was built mainly on vLLM, which is one of the engines it is ranked against. It also took ideas from SGLang and TensorRT-LLM. The release is licensed Apache 2.0.

    Source ↗
What it is
DeepSeek's own engine, released as open code for one DeepSeek model.
Its edge
Built together with a frontier model, targeting Nvidia SM100 and Huawei Ascend 950.
Who it suits
Suits those who run DeepSeek-V4.1-Flash offline; not anyone needing other models or a full service.
  • DeepSeek-R1 speed (tokens/s)25,000 tok/s per GPU (own figure)

Wikipedia · Wikidata

Why it is here

  • The launch repository lists 1 supported model, DeepSeek-V4.1-Flash, and 2 supported hardware families: NVIDIA SM100 and Huawei Ascend 950. That narrow scope fits its lower ranking on breadth and maturity. — (voice only)

    Source ↗
Did you know

  • DeepSeek's own speed figure comes from a split test. Prefill was measured on 4 Nvidia SM100 GPUs and decode on 64. The released code does not include a KV cache system or prefill/decode disaggregation.

    Source ↗
History

  • In 2026, DeepSeek opened the code for DeepInfer, an engine it said powers its large-scale production services. The public release was a limited window into that system: it offered offline inference for DeepSeek-V4.1-Flash and said several parts needed for a complete service were not included. — (voice only)

    Source ↗
Keep in mind

  • DeepSeek's speed numbers are its own and have not been independently checked. One write-up says it published no figures at all for Huawei Ascend 950, even though that chip is listed as supported.

    Source ↗
Where to go further

  • Read the README first. It says the project runs one model, DeepSeek-V4.1-Flash, on NVIDIA SM100 GPUs or Huawei Ascend 950 chips. Its setup starts with an empty Python 3.12 environment.

    Source ↗

Video: DeepSeek R1 Explained to your grandma · AI with Alex

Compare the top three ↓

The top three side by side

SGLangllama.cppDeepInfer (DeepSeek)
In the AIs' lists6 / 76 / 76 / 7
DeepSeek-R1 speed (tokens/s)3,276 tok/s at 200 prompts37 tok/s on 8 AMD MI300X25,000 tok/s per GPU (own figure)
Put first byGrok——

Places with the same number of AIs are ordered by their average place in the AIs' own lists.

More ratings

03Worth knowing

What the AIs noticed while building the list.

On the same 8× H200 DeepSeek-R1 setup the published winner flips between runs: vLLM 1,137 vs 1,041 tok/s, then SGLang 1,329 vs 1,042. Going from 50 to 200 prompts can flip it again.

Noticed by:
Included it: ChatGPT, Grok, Gemini, Kimi, Wenxin. Did not include: Claude, Meta.
Source ↗

'Non-Nvidia support' depends on where you draw the line: vLLM's Ascend and Gaudi support lives in plugin repos, while SGLang and llama.cpp keep Ascend in their main code.

Noticed by:
Included it: Claude, Grok, Gemini. Did not include: ChatGPT, Meta, Kimi, Wenxin.
Source ↗

Moving off Nvidia is already peer-reviewed on AMD: one MI355X Llama-2-70B Interactive result in MLPerf v6.0 was 73,608 tok/s against 70,724 for an 8-way B300 TensorRT submission.

Noticed by:
Included it: Grok. Did not include: Claude, ChatGPT, Meta, Gemini, Kimi, Wenxin.
Source ↗

Software alone moves the numbers: MLPerf v6.0 reports 2.7 times more DeepSeek-R1 throughput on the same Nvidia hardware than six months earlier.

Noticed by:
Included it: Claude. Did not include: ChatGPT, Meta, Grok, Gemini, Kimi, Wenxin.
Source ↗

DeepInfer's 155 trillion tokens a day covers one DeepSeek cluster and is self-reported. The public release is offline only, so cache, prefill/decode split and load balancing are left to the operator.

Noticed by:
Included it: Claude, ChatGPT, Grok, Kimi. Did not include: Meta, Gemini, Wenxin.
Source ↗

04Questions people ask

vLLM vs SGLang?

Neither wins everywhere: vLLM has the larger ecosystem and an audited MLPerf result, while SGLang keeps Ascend and AMD in its main code. On 8× H200 DeepSeek-R1, each has won one published run.

Source ↗
vLLM vs SGLang vs llama.cpp?

vLLM and SGLang are cluster servers. llama.cpp is one code tree covering Metal, HIP, SYCL, Vulkan and Huawei CANN, but on one A100 70B comparison it gave 900–1,300 tok/s against vLLM's 2,800–3,400.

Source ↗
vLLM vs SGLang performance?

At 200 prompts on 8× H200 DeepSeek-R1, SGLang hit 3,276 tok/s and vLLM 2,499. With fewer prompts vLLM led 1,137 to 1,041, so results depend on the setup.

Source ↗
vLLM vs SGLang vs Ollama?

Ollama is a local runner built on llama.cpp. One cited test has it at about 41 tok/s under load against about 793 for vLLM, so it suits desktops rather than serving many users.

Source ↗

05How we counted

In plain words, no formulas.

1The question

Top 10 Open AI Serving Engines: Speed Against Freedom From Nvidia

2Why this topic

The reader gets 'which is faster' and 'which frees me from one chip maker' side by side, where speed benchmarks and hardware support lists each show only one of them.

3Where it applies

global; open-source engines for serving large language models (vLLM, SGLang, DeepInfer, llama.cpp, TensorRT-LLM as the Nvidia reference, and others)

4What we looked at

  • Output throughput and time to first token on a stated model, hardware and concurrency level
  • Number of non-Nvidia chip families supported in the main code (AMD ROCm, Huawei Ascend, Intel Gaudi, Apple Metal, others) rather than in separate vendor forks
  • Which large open models run on each chip family with official support
  • Maturity: documentation, release pace and active maintainers
  • Michaelvll llm-ie-benchmarks — Output throughput of vLLM and SGLang on DeepSeek-R1, 8× Nvidia H200, at stated prompt and output lengths and prompt counts.
  • MLPerf Inference v6.0 (as read by mlai.qa) — Audited tokens/s for gpt-oss-120b on 8× B200: vLLM against TensorRT-LLM, offline and with a server latency cap.
  • vLLM Ascend release notes — Models and Atlas hardware covered by the separate vllm-ascend plugin (v0.30.0rc1, 11 Oct 2026).
  • SGLang README hardware table — Chip platforms the main SGLang project lists as supported.

5Who ranked

The leading AIs, each building its own list of ten independently and naming a source for every place: Claude, ChatGPT, Meta, Grok, Gemini, Kimi, Wenxin.

6How the places were set

The same entry written differently by different AIs was counted as one. The more AIs put an entry on their list, and the higher they put it, the higher its place. "AI agreement" shows how many of the AIs had it on their own list. Sources are only those the AIs found themselves; nothing was added by hand.

06Each AI's own list

🇺🇸 Claude10 places · 3 notes

The same question, answered by Claude alone. The final list above combines all the AIs.

  1. 1
    vLLM

    Its plugin model lets Ascend, Intel Gaudi, MetaX, Rebellions and IBM Spyre plug in, and Ascend's plugin was the reference case for it.

    Source ↗
  2. 2
    SGLang

    It treats Ascend as part of the shared model code, with _is_npu branches in the main files, and AMD, TPU and Gaudi routes on top.

    Source ↗
  3. 3
    llama.cpp

    The widest chip list of any engine here: Metal, CUDA, HIP, Vulkan, SYCL, CANN (Ascend) and MUSA in one tree, but built for single-user speed rather than datacenter batches.

  4. 4
    xLLM (JD, now OpenAtom)

    Built for Chinese accelerators and claimed at 1.7 times MindIE and 2.2 times vLLM-Ascend on Qwen under equal TPOT, which is the authors' own claim.

  5. 5
    DeepInfer (DeepSeek)

    Released one day ago and already described as running about 155 trillion tokens a day, but it supports one model on two chips.

  6. 6
    TensorRT-LLM (Nvidia reference)

    The speed reference at about 2,400 tok/s on H100 in one comparison, with zero non-Nvidia chips.

  7. 7
    Ollama

    Easiest to install, but one report has it flatlining near 41 tok/s under load while vLLM peaked near 793.

    Source ↗
  8. 8
    MLC LLM

    Aimed at edge, mobile and WebGPU, with about 21.6K GitHub stars.

    Source ↗
  9. 9
    Hugging Face TGI

    A production server with automatic quantization and LoRA, but I found no non-Nvidia or DeepSeek-R1 data.

  10. 10
    MindIE (Huawei)

    Official on Atlas 800 A2 and 300I Duo, but one benchmark write-up calls its setup threshold high.

    Source ↗

What Claude noticed

The 'non-Nvidia support' count depends on where you draw the line. vLLM's Ascend, Gaudi and MetaX support lives in separate plugin repos, while SGLang keeps its Ascend backend inside the main code and llama.cpp keeps nearly all backends in one tree.

Source ↗

On the same hardware, software updates alone raised throughput a lot: MLPerf v6.0 reports 2.7 times more DeepSeek-R1 throughput on the same NVIDIA hardware than six months earlier. SemiAnalysis's V4 tracking headline cites 100x improvement in 26 days.

Source ↗

AMD says SGLang is the default for DeepSeek-R1 on its chips and expects vLLM parity soon, so the winner there is not settled.

Source ↗
🇺🇸 ChatGPT10 places · 4 notes

The same question, answered by ChatGPT alone. The final list above combines all the AIs.

  1. 1
    vLLM (2026)

    A reproducible H200 run gives vLLM 1,137 output tok/s versus SGLang’s 1,041 at 1,000 input/2,000 output tokens—but another run reverses the winner.

    Source ↗
  2. 2
    SGLang (2026)

    At 200 prompts, SGLang leads vLLM by about 31% on the cited 1k/2k H200 workload; with fewer prompts, it loses.

    Source ↗
  3. 3
    llama.cpp (2026)

    One codebase reaches Apple Metal, AMD, Intel SYCL, Vulkan and other backends, making it the broadest practical hardware escape hatch here.

    Source ↗
  4. 4
    DeepInfer (2026)

    It pairs an unusually important Ascend 950 target with a launch-day codebase of only three public commits and no published Ascend throughput.

    Source ↗
  5. 5
    TensorRT-LLM (Nvidia reference, 2026)

    It remains the Nvidia-focused speed reference, but its value as a route away from Nvidia is effectively zero.

    Source ↗
  6. 6
    LMDeploy (2026)

    A substantial serving option with documented non-Nvidia ambitions, but the evidence gathered here does not put it on equal benchmark footing with vLLM and SGLang.

    Source ↗
  7. 7
    Hugging Face TGI (2026)

    A mature, documented deployment project, though the cited measurements do not establish a matching cross-chip speed comparison.

    Source ↗
  8. 8
    MLC-LLM (2026)

    Its compiler-oriented portability broadens device options, but this evidence set does not provide a same-model, same-hardware throughput comparison.

    Source ↗
  9. 9
    Ollama (2026)

    Its simple local distribution reaches many users, but user convenience and backend portability do not substitute for a controlled serving benchmark.

    Source ↗
  10. 10
    Aphrodite Engine (2026)

    A community serving alternative, but public evidence gathered here is thinner on cross-vendor support and comparable throughput than the leaders.

    Source ↗

What ChatGPT noticed

The published vLLM/SGLang contest is workload-sensitive: on 8×H200, changing the prompt count from 50 to 200 flips the winner, and the benchmark authors explicitly warn against reading one table as a universal verdict.

Source ↗

DeepInfer is a meaningful Nvidia-independence signal, but its Oct. 20 public code release is narrower than the production-service description: offline V4.1-Flash, two listed accelerator targets, and no public Ascend performance chart.

Source ↗

A broad backend list does not guarantee equal speed or model support. llama.cpp’s build guide lists several backends, while vLLM and SGLang publish more explicit per-model Ascend matrices.

Source ↗

MLPerf is a useful outside check for hardware-system results, but its inference submissions do not create a like-for-like ranking of every open serving engine on one model and configuration.

Source ↗
🇺🇸 Grok10 places · 3 notes

The same question, answered by Grok alone. The final list above combines all the AIs.

  1. 1
    SGLang

    One README names AMD, TPU, Intel, Apple and Huawei next to Nvidia — and one H100 test has it at 16,215 tokens/s against vLLM's 12,553.

    Source ↗
  2. 2
    vLLM

    On the one audited table, vLLM beat TensorRT-LLM by 8% when the goal was raw throughput, and lost by 22% when the goal was the latency cap.

    Source ↗
  3. 3
    llama.cpp

    CANN, HIP, Metal, Vulkan and SYCL sit in one tree — the same tree that, on one A100 write-up, serves a 70B model at roughly a third of vLLM's batch rate.

  4. 4
    DeepInfer

    Day-old, 14.1k stars, two chip families in one repo — and it runs one model, with the cache left as homework.

  5. 5
    TensorRT-LLM

    The Nvidia yardstick: 87,444 tokens/s under MLPerf's latency cap, and zero non-Nvidia chip families.

    Source ↗
  6. 6
    Ollama

    184,000 stars and a one-line install — it is llama.cpp in a friendlier coat, not a second answer to a full GPU cluster.

    Source ↗
  7. 7
    vLLM Ascend

    The 11 Oct candidate already names DeepSeek V4.1, GLM-5.3 and Qwen3.8-27B — and it is a second repository, not a flag inside vLLM.

    Source ↗
  8. 8
    Text Generation Inference (Hugging Face)

    Still the third name on a 70B bake-off: about 2,100–2,600 tokens/s, behind vLLM's 2,800–3,400 and ahead of llama.cpp.

    Source ↗
  9. 9
    llamafile

    One file, any major desktop OS — Mozilla's packing of llama.cpp, which is why it inherits the chip list and also the speed ceiling.

    Source ↗
  10. 10
    Triton Inference Server

    The open shell around TensorRT-LLM: a real server, a BSD licence, and still an Nvidia address.

    Source ↗

What Grok noticed

The two careful DeepSeek-R1 runs on the same 8× H200 box swap the winner. vLLM leads the first published pair; SGLang leads the rerun three weeks of versions later. The collector's note: going from 50 prompts to 200 can flip the verdict.

Source ↗

DeepInfer's 155 trillion tokens a day is one cluster of DeepSeek-V4.1-Flash, not the whole company, and the release is offline only. The parts that make it a service — cache, splitting prefill from decode, load balancing — stay on the operator.

Source ↗

Leaving Nvidia is already peer-reviewed on AMD. In MLPerf v6.0 every MI355X and MI350X result used PyTorch/ROCm, several naming vLLM, and one MI355X Llama-2-70B Interactive result was 73,608 tokens/s against 70,724 for an 8-way B300 TensorRT submission.

Source ↗
🇺🇸 Gemini10 places · 3 notes

The same question, answered by Gemini alone. The final list above combines all the AIs.

  1. 1
    vLLM (2026)

    The undisputed backbone of production AI, natively unifying Nvidia CUDA, AMD ROCm, Huawei Ascend, and Intel Gaudi under a single continuous batching engine.

    Source ↗
  2. 2
    SGLang (2026)

    Crushes agentic and multi-turn latency via RadixAttention, outpacing standard runtimes by up to 3x on shared prefix workloads while offering native ROCm support.

    Source ↗
  3. 3
    llama.cpp (2026)

    The ultimate portable escape hatch from enterprise GPU monopolies, serving 70B models seamlessly across Apple Metal, CPU AVX-512, and ROCm with zero CUDA dependencies.

    Source ↗
  4. 4
    DeepInfer (2026)

    DeepSeek's battle-tested 155-trillion-token-a-day engine, built with custom FlashMLA and DeepGEMM kernels targeting Nvidia SM100 and Huawei Ascend 950.

    Source ↗
  5. 5
    TensorRT-LLM (Nvidia Reference, 2026)

    Nvidia's proprietary speed crown: unmatched raw throughput on H100 and Blackwell, but completely locked to the Nvidia hardware ecosystem.

    Source ↗
  6. 6
    LMDeploy (2026)

    Shanghai AI Lab's lightweight powerhouse, achieving lightning-fast TurboMind C++ token generation and official Huawei Ascend acceleration.

    Source ↗
  7. 7
    TGI - Text Generation Inference (2026)

    Hugging Face's hardened enterprise server, providing solid multi-accelerator deployment across Intel Gaudi, AMD ROCm, and Nvidia GPUs.

    Source ↗
  8. 8
    Ollama (2026)

    Packaging llama.cpp and custom runners into a single CLI command, turning local non-Nvidia workstations into plug-and-play AI servers.

    Source ↗
  9. 9
    Aphrodite Engine (2026)

    PygmalionAI's community engine focusing on large unconstrained MoE inference across AMD ROCm and diverse consumer hardware.

    Source ↗
  10. 10
    MLC-LLM (2026)

    Compiles deep learning models into standalone executable machine code via Apache TVM, running on Qualcomm, Apple Metal, and Vulkan devices.

    Source ↗

What Gemini noticed

The battle between vLLM and SGLang highlights a fundamental design split: vLLM dominates general-purpose batch throughput and broad hardware support, while SGLang's RadixAttention provides massive throughput gains on shared-prefix agentic tasks.

Source ↗

True freedom from Nvidia depends heavily on unified upstream codebases: while AMD ROCm has achieved first-class native status in vLLM and SGLang, Huawei Ascend still frequently requires specialized plugins (vllm-ascend) or vendor forks.

Source ↗

DeepInfer's open release of high-performance MoE kernels (FlashMLA, DeepGEMM, DeepEP) provides the first hyperscale production blueprint for running 600B+ parameter models on Huawei Ascend 950 NPUs alongside Nvidia SM100.

Source ↗
🇨🇳 Kimi10 places · 4 notes

The same question, answered by Kimi alone. The final list above combines all the AIs.

  1. 1
    vLLM (PyTorch Foundation)

    ~91,000 stars, releases every few weeks (v0.30.0, Sep 22), and NVIDIA+AMD+Intel support in main code with Gaudi/Ascend/TPU arriving as plugins - the default escape route from Nvidia.

    Source ↗
  2. 2
    SGLang (RadixArk / LMSYS origins)

    RadixAttention makes it the agent-loop specialist, and its Ascend port officially supports more frontier open models - DeepSeek V4.1, GLM-5.3, Kimi K3, MiniMax-M3, Hy4 - than any other engine.

    Source ↗
  3. 3
    llama.cpp (ggml)

    One C++ codebase carrying CUDA, ROCm, Vulkan, Metal, SYCL and MUSA backends - it runs the full 671B DeepSeek-R1 on 8 AMD MI300X GPUs at ~37 output tok/s with no vendor fork.

    Source ↗
  4. 4
    TensorRT-LLM (NVIDIA)

    The speed king on Nvidia - 2,780 tok/s on Llama-3.3-70B at 100 concurrent requests vs vLLM's 2,400 - paid for with a 28-minute engine compile and zero non-Nvidia support.

    Source ↗
  5. 5
    DeepInfer (deepseek-ai)

    One day old and already the only engine co-designed with a frontier model: DeepSeek's own production stack, tuned for V4.1-Flash on Nvidia SM100 and Huawei Ascend 950 from release.

    Source ↗
  6. 6
    KTransformers (KVCache.AI)

    The academic outlier whose CPU-offload design (SOSP'25 paper) runs 671B-class MoE models on 24GB-class GPU memory - and whose kernels now live inside SGLang.

    Source ↗
  7. 7
    Ollama

    The developer on-ramp: one binary wrapping llama.cpp across CUDA, ROCm and Metal - and since 0.19 a preview MLX backend that is 1.6-2x faster at decode on Apple Silicon.

    Source ↗
  8. 8
    LMDeploy / TurboMind (InternLM)

    The C++ engine that beats vLLM on its own turf - 1,980 vs 1,850 tok/s on a 7B H100 - while its PyTorch engine adds official Huawei Ascend support the TurboMind core lacks.

    Source ↗
  9. 9
    MLX / mlx-lm (Apple)

    Apple's MIT-licensed array framework is the only serious engine native to unified memory - vllm-mlx hits 525 tok/s on an M4 Max, 21-87% above llama.cpp.

    Source ↗
  10. 10
    Hugging Face TGI (text-generation-inference)

    Now in maintenance mode, Hugging Face's own Rust server tells users to move to vLLM or SGLang - the formal sign that the serving stack consolidated around two Apache-2.0 leaders.

    Source ↗

What Kimi noticed

The same benchmark flips with the runner: on identical 8xH200 DeepSeek-R1 hardware, vLLM's script shows vLLM 1137 vs SGLang 1041 tok/s; SGLang's script shows SGLang 1329 vs vLLM 1042; and raising 50 to 200 prompts flips it again (SGLang 3276 vs vLLM 2499) - 'vLLM vs SGLang' has no setup-independent answer.

Source ↗

llama.cpp is the great equalizer: AMD's own blog runs the full 671B DeepSeek-R1 on 8 MI300X GPUs at 37 tok/s decode with zero fork - the only stack where a frontier open model runs on non-Nvidia silicon day one, which is why Ollama, LM Studio and KTransformers all build on it.

Source ↗

DeepInfer is the first time a frontier lab's production engine treats Huawei Ascend 950 as a first-class, day-zero backend - and ships the whole cluster recipe ($38.5M capex) to reproduce it; it is the strongest 'freedom from Nvidia' statement of 2026 despite being 24 hours old.

Source ↗

The field has consolidated and bifurcated: TensorRT-LLM stays the Nvidia-only speed reference (with a 28-minute compile), while TGI's maintenance-mode notice and SGLang's RadixArk spinout ($100M seed, NVIDIA and AMD both investing) show the open-serving center of gravity settling on vLLM and SGLang.

Source ↗
🇨🇳 Wenxin10 places · 1 notes

The same question, answered by Wenxin alone. The final list above combines all the AIs.

  1. 1
    vLLM (global, 2023-ongoing)

    The default for 70% of open-model deployments — and now runs natively on Huawei Ascend too

    Source ↗
  2. 2
    SGLang (global, 2024-ongoing)

    Faster than vLLM in head-to-head on the same GPUs — and beat it to Ascend support

    Source ↗
  3. 3
    DeepInfer (DeepSeek, China, 2026)

    Born free on Oct 20 2026 — runs DeepSeek's own 155T-token workload on Nvidia and Ascend alike

    Source ↗
  4. 4
    llama.cpp (global, 2023-ongoing)

    Runs 70B models on a single MacBook — the people's engine, even if it's not the fastest

    Source ↗
  5. 5
    TensorRT-LLM (Nvidia, 2023-ongoing)

    Nvidia's own speed king — fastest on H100/B200 but useless if you want off-Nvidia

    Source ↗
  6. 6
    Ollama (global, 2023-ongoing)

    Not a raw engine — a polished wrapper on llama.cpp that made local AI mainstream for non-engineers

    Source ↗
  7. 7
    TGI / Text Generation Inference (Hugging Face, 2022-ongoing)

    HuggingFace's production-grade wrapper — widely used on cloud but frozen on chip diversity

    Source ↗
  8. 8
    OpenVINO / OpenVINO GenAI (Intel, 2023-ongoing)

    Intel's answer for Gaudi and CPU — fast on Intel iron, but model support lags behind

    Source ↗
  9. 9
    TabbyML / Tabby (global, 2023-ongoing)

    Growing niche engine that adds Ascend and AMD more seriously than the big names

    Source ↗
  10. 10
    PowerInfer (XiaoMi, China, 2024-ongoing)

    Mi's budget-server engine — runs 70B on one GPU but not battle-tested on Ascend yet

What Wenxin noticed

DeepInfer's release on Oct 20 2026 made it the only engine from a top-10 Chinese lab that is both open-source AND officially supports Huawei Ascend in main code — but it only runs DeepSeek models so far, limiting its generality.

Source ↗