vLLM vs TensorRT-LLM vs SGLang - Choosing a Self-Hosted Serving Engine

Managed vs. reserved vs. self-hosted inference covers the payment-model question — pay-as-you-go, reserved capacity, or buying and running the GPUs yourself. This page starts one decision downstream of that: self-hosting is already the constraint-driven choice, and the question is which serving engine actually runs on the hardware. vLLM, NVIDIA TensorRT-LLM, and SGLang are the three engines that dominate that choice in 2026. All three are Apache 2.0, all three expose an OpenAI-compatible HTTP API, and all three build on the same underlying idea — a paged or tree-structured KV cache plus continuous batching to keep a GPU saturated with concurrent requests. The differences that actually decide between them are not feature-row differences. They are a hardware gate, an operational-capability tradeoff, and a support-contract question — checked in that order, before any tokens-per-second number is worth reading.

Constraints that decide this before throughput does

Vendor and hardware lock-in — a gate for two of three, if you have this constraint at all

TensorRT-LLM runs on NVIDIA GPUs exclusively. Its own support matrix lists Blackwell, Hopper, Ada Lovelace, Ampere, and the Grace Hopper and GB200 NVL72 superchips as the architectures NVIDIA develops and tests against; anything not on that list gets community support only, by NVIDIA’s own statement.¹ There is no AMD, TPU, or Trainium path — none exists, and none is on a public roadmap.

vLLM and SGLang both target a materially wider surface. Per the PyTorch Foundation’s own announcement when it took vLLM on as a hosted project, vLLM runs on “NVIDIA GPUs, AMD GPUs, Google Cloud TPUs, Intel GPUs, Intel CPUs, Intel Gaudi HPUs, and AWS Neuron, among others,” plus ARM and IBM Z, with a plugin system that has already brought in IBM Spyre and Huawei Ascend as third-party backends.² SGLang’s own install documentation lists NVIDIA, AMD ROCm (including the MI300 and MI355 lines), Apple Metal, Intel Xeon CPUs and XPU, Google TPU, Ascend NPUs, and NVIDIA’s Jetson and DGX Spark as separately documented targets.³

This is a gate, not a tradeoff, for a specific and checkable reason: if an organization’s GPU fleet includes or is planned to include AMD Instinct, Google TPU, or AWS’s own silicon — or if hardware-vendor optionality is itself the point, so a future renegotiation with a GPU supplier stays credible — TensorRT-LLM is not a live option, independent of how it benchmarks on the hardware it does support. If the fleet is committed to NVIDIA and staying that way, this constraint eliminates nothing, and the choice moves to the tradeoffs below. Worth naming separately: because vLLM and SGLang both consume standard Hugging Face checkpoints through the same OpenAI-compatible API shape, switching between those two carries comparatively low technical lock-in. The sharp lock-in line in this comparison is NVIDIA-exclusivity, not a vLLM-versus-SGLang split.

Internal capability and operational complexity — a tradeoff that changed materially in 2026

vLLM installs with pip install vllm and serves a model behind an OpenAI-compatible endpoint with one command; it is widely treated as the easiest of the three to stand up without a dedicated ML-systems specialist on the team, and its scale of community contribution (see the resilience section below) tends to translate into same-day or next-day support for newly released open-weight models. SGLang’s operational shape is similar — a single sglang.launch_server command — with the added complexity that getting the most out of RadixAttention means understanding its cache-eviction behavior under memory pressure, not just launching a server.

TensorRT-LLM’s reputation for a heavier operational lift is real history, but the specific mechanism behind it changed in 2026 in a way worth verifying directly rather than assuming from older material. Through 2025, the standard TensorRT-LLM workflow compiled a model into a fixed .engine artifact with the trtllm-build CLI — an artifact tied to a specific model, precision, batch configuration, and GPU architecture, that had to be rebuilt to move to a different GPU generation. NVIDIA’s own release notes record a two-step transition away from that: version 1.0 made a PyTorch-native backend “the default LLM backend,” and version 1.2 — the current stable release as of this writing — removed the legacy ahead-of-time TensorRT backend entirely, in NVIDIA’s own words: “TensorRT backend removed. PyTorch is now the sole execution backend,” with trtllm-build, TrtLlmArgs, and the per-model convert_checkpoint.py scripts all deleted from the codebase.⁴ The current Quick Start guide confirms the practical effect: trtllm-serve "TinyLlama/TinyLlama-1.1B-Chat-v1.0" loads a Hugging Face checkpoint directly, with no separate build step exposed to the operator, cutting cold start from the old workflow’s tens of minutes to roughly 60–90 seconds by third-party reporting.⁵ Treat any claim that TensorRT-LLM still requires manually recompiling an engine per GPU as describing a workflow NVIDIA’s own mainline project has since retired — verify against the current docs, not the 2023–2024-era public description of the tool, before budgeting operational time around it. What hasn’t changed: TensorRT-LLM’s published “Quick Start Recipes” for specific large models are still scoped per GPU family — Blackwell and Hopper get first-class, NVIDIA-validated recipes; older architectures get comparatively less attention — so GPU-generation-specific tuning still matters for reaching NVIDIA’s own peak-throughput numbers, it’s just no longer a manual artifact-rebuilding step in the critical path.

Cost structure — not licensing, but engineering time against tokens per dollar

All three are free to run; there is no license fee differentiating them. The real cost differences show up in tokens-per-second per GPU-hour, which is exactly the “utilization decides everything” math worked out in self-hosting LLMs: hardware and economics — that page attributes a 25–75x swing in cost per million tokens on the same rented GPU to how well the serving stack keeps it saturated, not to the hardware itself. Serving-engine choice is one of the largest levers on that number. The frequent claim that TensorRT-LLM has the highest raw throughput “when tuned” carries a real asterisk: the tuning is engineering time — kernel selection, batch and quantization configuration validated for your specific model-and-GPU pairing — which the headline tokens-per-second figure does not itself show as a cost. Weigh a stated throughput number against who on the team can actually extract it, not just whether it’s the highest number on a slide.

Resilience and model-day-0 coverage — a tradeoff with a real continuity cost

If a roadmap depends on serving whatever open-weight model ships next — DeepSeek, Qwen, Llama, Kimi, and similar — community size is a rough but checkable proxy for how quickly a new architecture lands. As of this writing, vLLM’s GitHub repository carries roughly 91,000 stars, SGLang roughly 34,000, and TensorRT-LLM roughly 14,500.⁶ Stars are not a usage metric, but the gap tracks a real operational difference: vLLM and SGLang’s model support is contributed by a large, distributed community and tends to land within days of a new open-weight release, while TensorRT-LLM’s per-model support ships against NVIDIA’s own validated-model table on NVIDIA’s release cadence — narrower, and not driven by the same community throughput.

Contractual and vendor-support requirements — a gate for orgs that need a named support contract

TensorRT-LLM sits inside NVIDIA AI Enterprise, a paid support layer licensed per GPU socket that adds business-hours access to NVIDIA’s own engineers, priority notification of security fixes, and long-term-support API branches on top of the open-source project.⁷ vLLM’s largest commercial contributor, Neural Magic, was acquired by Red Hat in a deal that closed in January 2025, and Red Hat has since folded vLLM into the supported Red Hat AI stack.⁸ SGLang remains maintained by the non-profit LMSYS, but a commercial spinout — RadixArk, founded by contributors to the project — emerged in January 2026 at a reported $400 million valuation, explicitly to sell services around SGLang while keeping the core framework open source.⁹ If a procurement process specifically requires a named vendor and an SLA rather than an open-source repository with contributors, note the maturity gap here plainly: NVIDIA AI Enterprise and Red Hat both carry years of enterprise-support history, where RadixArk is, as of this writing, a 2026-vintage entity. That gap is a fact this page can report, not one it can resolve for a given contract.

What doesn’t differentiate here

Regulatory and jurisdictional exposure, and data gravity, are both resolved upstream of this page — by the decision to self-host at all, covered in managed vs. reserved vs. self-hosted inference and the methodology behind this wiki’s comparisons . None of these three engines changes where the hardware physically sits or who operates it — that was already decided before this comparison became relevant. If jurisdiction or data residency is still an open question, resolve it against the linked pages first; nothing below changes based on it.

What each one actually is

vLLM originated at UC Berkeley’s Sky Computing Lab, built around PagedAttention — a KV-cache scheme borrowed from operating-system virtual memory paging — published at SOSP 2023 and open-sourced the same year.¹⁰ It became a PyTorch Foundation-hosted project in 2025, and its current release is v0.28.0.¹¹

NVIDIA TensorRT-LLM is NVIDIA’s own optimized inference library, built on top of the proprietary TensorRT SDK (governed by NVIDIA’s separate software license agreement, distinct from the Apache 2.0 license on the TensorRT-LLM code itself) and CUDA.¹² Its current stable release is v1.2.1, with v1.3 in release candidates as of late August 2026.¹³

SGLang was introduced in January 2024 by researchers from Stanford, UC Berkeley, Texas A&M, and Shanghai Jiao Tong University, built around RadixAttention — a radix-tree structure for automatically reusing cached KV state across requests that share a prefix — and published at NeurIPS 2024.¹⁴ It is maintained by the non-profit LMSYS, with its current release at v0.5.18.¹⁵

The mechanism that actually differs: three answers to the same memory problem

All three engines solve the same underlying constraint — the KV cache is the thing that fills GPU memory during serving, and how it’s stored and reused determines both how many concurrent requests fit and how much recomputation happens. vLLM’s PagedAttention borrows OS-style memory paging to store the cache in fixed, non-contiguous blocks, eliminating fragmentation; vLLM also ships Automatic Prefix Caching on top of that same paged design, hashing each cache block by its tokens so that requests sharing an exact prefix reuse the cached blocks via a flat hash table.¹⁶ SGLang’s RadixAttention solves a related but distinct problem: instead of a flat hash table, it organizes cached prefixes in a radix tree, which is purpose-built for programs that branch into multiple continuations from a shared prefix — few-shot examples with several completions sampled per example, tree-of-thought exploration, or an agent loop that resends the same system prompt and tool definitions on every step. That’s the concrete version of “particularly strong for agentic and structured-generation workloads”: the advantage isn’t caching a prefix at all, which both engines now do, it’s efficiently caching branching reuse patterns, which is what agent and tool-calling loops produce. TensorRT-LLM’s paged KV cache does the same fragmentation-avoidance job as PagedAttention, layered underneath NVIDIA’s own hand-tuned CUDA kernels for attention and matrix multiplication — its advantage is squarely at the kernel level, not the cache-reuse-topology level the other two compete on.

On the throughput numbers, and why this page won’t rank them

Third-party technical benchmarks from 2026 disagree with each other on direction, which is itself the most honest thing to report. One May 2026 comparison found vLLM leading raw throughput on Qwen2.5-7B on a single H100 (23,500 tok/s against TensorRT-LLM’s 16,500 and SGLang’s 16,800, with SGLang far ahead on time-to-first-token at 1.8 seconds against TensorRT-LLM’s 7.8), while showing all three within a few percent of each other on a longer-context Qwen3-32B workload across two H100s.¹⁷ A separate mid-2026 comparison on Llama-3.3-70B at FP8 found TensorRT-LLM ahead of vLLM by 8–13% at every concurrency level tested, with SGLang leading specifically on time-to-first-token for prefix-heavy traffic — the opposite ranking on raw throughput from the first source, on a different model size and precision.¹⁸ A third found SGLang roughly 29% ahead of vLLM on prefix-heavy Llama-3.1-8B traffic specifically.¹⁹ None of these are the vendors’ own marketing pages — they’re independent technical blogs with published methodology and dates — but they still disagree with each other by double-digit percentages depending on model size, precision, and concurrency, which is consistent with recent academic work showing that common LLM-serving benchmark tooling itself introduces measurement bias under concurrency that can distort exactly these kinds of comparisons.²⁰ Treat every number in this section as a snapshot of one workload on one engine version, not a ranking — the only trustworthy throughput number for a specific deployment decision is one measured against that deployment’s own model, quantization, and traffic shape.

The comparison

What this comparison cannot resolve

The actual throughput and latency for a specific model, quantization, hardware generation, and concurrency profile has to be measured against that workload — the section above shows why no published number, from any source, substitutes for that. Whether an existing team’s CUDA and ML-systems expertise offsets TensorRT-LLM’s remaining GPU-family-specific tuning surface is a team-specific judgment, not a general one. The actual pricing, SLA terms, and contract structure behind NVIDIA AI Enterprise, Red Hat’s vLLM support, or RadixArk’s commercial offering are not fully public and require a direct sales conversation with each — this page reports that the options differ in maturity, not what any of them costs. And whether a given organization’s procurement or security-review process treats “an Apache 2.0 project with a corporate contributor” the same as “a vendor product with a support contract” is a policy question specific to that organization, not one a general comparison can settle.

Further reading

• Managed vs. reserved vs. self-hosted inference : the payment-model decision this page assumes has already been made.
• Constraint-driven comparisons : the methodology behind separating gates from tradeoffs used throughout this page.
• Self-hosting LLMs: hardware and economics : the utilization math behind why serving-engine throughput matters as much as GPU choice.
• GPU clouds and neoclouds compared : where to rent the hardware any of these three engines runs on.
• KV cache : the memory structure PagedAttention, RadixAttention, and TensorRT-LLM’s paged cache all manage differently.
• Continuous batching : the request-scheduling technique all three engines share underneath their different cache strategies.
• vLLM , NVIDIA TensorRT-LLM , SGLang : this wiki’s dedicated pages for each engine.

Sources

1. NVIDIA, TensorRT-LLM Support Matrix: https://nvidia.github.io/TensorRT-LLM/reference/support-matrix.html — supported GPU architectures and the community-support-only statement for unlisted architectures.
2. PyTorch Foundation, “PyTorch Foundation Welcomes vLLM as a Hosted Project”: https://pytorch.org/blog/pytorch-foundation-welcomes-vllm/ — hardware backend list and Foundation-hosted governance status.
3. SGLang documentation, installation guide: https://docs.sglang.io/get_started/install.html — per-platform installation pages for NVIDIA, AMD, Apple Metal, Intel, TPU, and Ascend.
4. NVIDIA, TensorRT-LLM Release Notes: https://nvidia.github.io/TensorRT-LLM/release-notes.html — the v1.0 default-backend change and the v1.2 removal of the legacy TensorRT (ahead-of-time engine) backend.
5. NVIDIA, TensorRT-LLM Quick Start Guide: https://nvidia.github.io/TensorRT-LLM/quick-start-guide.html — direct trtllm-serve invocation against a Hugging Face model identifier with no separate build step; cold-start timing reported by RunPod, “vLLM vs TensorRT-LLM: Which Inference Engine to Deploy”: https://www.runpod.io/articles/comparison/vllm-vs-tensorrt-llm (published June 23, 2026, updated August 25, 2026).
6. GitHub repository metadata for vllm-project/vllm, NVIDIA/TensorRT-LLM, and sgl-project/sglang, retrieved September 2026 via the GitHub API — star counts as a rough, checkable proxy for community size, not a usage metric.
7. NVIDIA, NVIDIA AI Enterprise documentation: https://docs.nvidia.com/ai-enterprise/index.html — per-GPU-socket licensing, business-support terms, and long-term-support branches covering TensorRT-LLM among other included software.
8. Red Hat, “FAQ: Red Hat to acquire Neural Magic”: https://www.redhat.com/en/blog/red-hat-acquire-neural-magic-FAQ ; Business Wire, “Red Hat Completes Acquisition of Neural Magic” (January 13, 2025): https://www.businesswire.com/news/home/20250113192550/en/Red-Hat-Completes-Acquisition-of-Neural-Magic-to-Fuel-Optimized-Generative-AI-Innovation-Across-the-Hybrid-Cloud — acquisition close date and vLLM’s folding into Red Hat AI.
9. Wikipedia, “SGLang”: https://en.wikipedia.org/wiki/SGLang — RadixArk’s emergence in January 2026 and reported $400 million valuation, sourced there to contemporaneous reporting; treat as a recent and still-developing fact to reverify directly with RadixArk before relying on specific contract terms.
10. Kwon, W. et al., “Efficient Memory Management for Large Language Model Serving with PagedAttention,” SOSP 2023: https://arxiv.org/abs/2309.06180 — PagedAttention design and origin at UC Berkeley.
11. GitHub, vllm-project/vllm releases: https://github.com/vllm-project/vllm/releases — current release version, retrieved via the GitHub API, September 2026.
12. GitHub, NVIDIA/TensorRT-LLM, LICENSE file: https://github.com/NVIDIA/TensorRT-LLM/blob/main/LICENSE — Apache 2.0 license text for the TensorRT-LLM codebase; NVIDIA, TensorRT Software License Agreement: https://docs.nvidia.com/deeplearning/tensorrt/latest/reference/sla.html — the separate proprietary license governing the underlying TensorRT SDK dependency.
13. GitHub, NVIDIA/TensorRT-LLM releases: https://github.com/NVIDIA/TensorRT-LLM/releases — current stable (v1.2.1) and release-candidate (v1.3.0rc) versions, retrieved via the GitHub API, September 2026.
14. Zheng, L. et al., “SGLang: Efficient Execution of Structured Language Model Programs,” NeurIPS 2024: https://arxiv.org/abs/2312.07104 — RadixAttention design, author institutions, and the paper’s own throughput claims against 2024-era baselines (treat as historical evidence for the approach, not a current benchmark against 2026 engine versions).
15. GitHub, sgl-project/sglang releases: https://github.com/sgl-project/sglang/releases — current release version, retrieved via the GitHub API, September 2026; SGLang GitHub repository README: https://github.com/sgl-project/sglang — LMSYS maintainership and Apache 2.0 license.
16. vLLM documentation, “Automatic Prefix Caching”: https://docs.vllm.ai/en/latest/design/automatic_prefix_caching.html — hash-table-based prefix cache design and its equivalence to a radix-tree eviction policy under full attention.
17. Jarvislabs, “vLLM, SGLang, or TensorRT-LLM? Picking an LLM Serving Stack” (May 18, 2026): https://jarvislabs.ai/blog/vllm-sglang-trtllm-comparison — Qwen2.5-7B and Qwen3-32B throughput and latency figures.
18. Spheron Network, “vLLM vs TensorRT-LLM vs SGLang: Which Is Fastest? (H100 Benchmarks)”: https://www.spheron.network/blog/vllm-vs-tensorrt-llm-vs-sglang-benchmarks/ — Llama-3.3-70B FP8 concurrency-level throughput comparison.
19. RunPod, “vLLM vs TensorRT-LLM: Which Inference Engine to Deploy” (published June 23, 2026, updated August 25, 2026): https://www.runpod.io/articles/comparison/vllm-vs-tensorrt-llm — prefix-heavy Llama-3.1-8B SGLang-vs-vLLM throughput figures and the TensorRT-LLM per-architecture engine history.
20. “Identifying and Mitigating Systemic Measurement Bias in Production LLM Inference Benchmarks,” arXiv:2605.24217: https://arxiv.org/abs/2605.24217 — client-side queuing bias in common LLM-serving benchmark tooling under concurrency, and a proposed corrected methodology.