Inference Servingvllm

Why vLLM's OpenAI-compatible API returns 404, and how to fix the request

Error
{"object":"error","message":"The model `gpt-3.5-turbo` does not exist.","type":"NotFoundError","code":404}

Also appears as

  • 404 Not Found calling /chat/completions (missing the /v1 prefix)

Short answer

This 404 is almost always a client-side mismatch, not a server bug: either the request hit the wrong route, such as missing the /v1 prefix, or the model field in the request body doesn't match the exact served-model-name (or default model repo id) vLLM registered at startup. Fix the URL and model name to match what /v1/models actually reports.

Affects: vLLM 0.3 and later, any deployment; also applies to other OpenAI-compatible servers with equivalent model-name or route mismatches

Fix it in under a minute

  1. 1Call GET /v1/models against your server and copy the exact id value it returns.
  2. 2Use that exact string as the model field in your request body, character for character.
  3. 3Confirm your request URL includes the /v1 prefix, for example http://host:8000/v1/chat/completions, not just /chat/completions.
  4. 4If you set --served-model-name at launch, that alias, not the Hugging Face repo id, is what clients must send.
  5. 5Re-test with curl before touching client SDK code, to isolate whether the problem is the server or the client library.

How to confirm this is your problem

  • Server is up and /v1/models returns a valid list, but chat completions requests 404.
  • Works with curl using the exact model id but fails from an application using a hardcoded model name like gpt-3.5-turbo or gpt-4.
  • Switching between --served-model-name and the raw Hugging Face repo id changes whether requests succeed.
  • Client library defaults to a path that omits the /v1 prefix vLLM expects.

Root causes and fixes

Most common

Request's model field doesn't match the server's registered model name

Many OpenAI-SDK-based clients default to a model string like gpt-3.5-turbo, hardcoded for a previous provider. vLLM only recognizes the exact Hugging Face repo id it loaded, or the alias given with served-model-name, and returns 404 for anything else because from its perspective that model simply isn't loaded.

Fix: Set the client's model parameter to exactly match the id returned by GET /v1/models, or launch vLLM with --served-model-name set to the friendly name your client already expects.

Commands
curl http://localhost:8000/v1/models
vllm serve MODEL_ID --served-model-name gpt-3.5-turbo
Common

Missing /v1 prefix in the request path

vLLM's OpenAI-compatible server namespaces every endpoint under /v1, mirroring OpenAI's own API. A request to /chat/completions instead of /v1/chat/completions hits no registered route at all, and the ASGI framework returns a generic 404, not a vLLM-specific error.

Fix: Point your base_url, or the OpenAI client's base URL config, at http://host:port/v1, not just http://host:port.

Commands
curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d "{\"model\":\"MODEL_ID\",\"messages\":[{\"role\":\"user\",\"content\":\"hi\"}]}"
Common

Hitting an endpoint vLLM doesn't implement for the version you're running

Not every OpenAI API surface is implemented in every vLLM release. Calling an endpoint like /v1/embeddings or an audio endpoint on a version or model that doesn't support it returns 404 even though other endpoints work fine on the same server.

Fix: Check the vLLM release notes and docs for your installed version to confirm the endpoint and model type you're calling are actually supported.

Occasional

Load balancer or reverse proxy stripping or rewriting the /v1 path

An nginx, API gateway, or ingress rule that rewrites paths, common when proxying multiple backend services under one domain, can accidentally strip the /v1 segment before the request reaches vLLM, producing a 404 that looks like a client bug but is actually a proxy misconfiguration.

Fix: Curl the vLLM server directly, bypassing the proxy, to confirm the same request succeeds, then fix the proxy's path rewrite rule.

Commands
curl http://vllm-pod-ip:8000/v1/models
Rare

Client SDK version expects a route vLLM doesn't expose

Newer OpenAI SDK versions add new endpoints, such as a responses API, that vLLM's OpenAI-compatibility layer may not implement yet, so calls using the newest SDK conventions 404 even though older chat completions calls work fine.

Fix: Use the chat completions or completions client methods explicitly rather than a newer SDK convenience method vLLM doesn't yet mirror.

Diagnostic commands

List models the server actually has loaded

curl http://localhost:8000/v1/models

The id field(s) returned are the only valid values for the model parameter in requests; anything else 404s.

Test the canonical chat completions route directly with curl

curl -s http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d "{\"model\":\"MODEL_ID\",\"messages\":[{\"role\":\"user\",\"content\":\"hi\"}]}"

If this succeeds but your app still 404s, the bug is in the client's base URL or model name, not the server.

Confirm no proxy is rewriting the path

curl http://VLLM_INTERNAL_IP:8000/v1/models

Comparing the direct-to-pod result against the result through your load balancer or ingress isolates whether a reverse proxy is the culprit.

Stopping it from happening again

  • Standardize on setting --served-model-name explicitly at launch so client code never has to track raw Hugging Face repo ids.
  • Add an automated smoke test that hits /v1/models and /v1/chat/completions after every deployment before routing real traffic.
  • Document the exact base URL, including /v1, and model name in onboarding docs for any team consuming the endpoint.
  • Version-pin your OpenAI client SDK and test against vLLM's supported endpoint surface before upgrading either side.

When this becomes an architecture problem

If you're running many models behind one gateway and constantly hitting name-mismatch 404s across teams, that's a routing and naming-convention problem worth solving with a model registry or API gateway layer rather than repeatedly fixing individual client configs.

Frequently asked questions

Why does vLLM care about the exact model name if it only has one model loaded?

It mirrors OpenAI's multi-model API contract on purpose, so existing OpenAI-client code, including load balancers routing by model name, works unmodified. That fidelity means it validates the model field strictly rather than ignoring it, which is why an unset or wrong default 404s.

Can I make vLLM accept any model name so my app doesn't need changes?

Yes, launch it with --served-model-name set to whatever string your application already sends, such as gpt-3.5-turbo, and vLLM will register that as the accepted model id even though it's serving a completely different underlying model.

Is a 404 always the client's fault?

Usually yes for this specific error, but always confirm by curling the server directly first. If a reverse proxy or gateway sits in front of vLLM, path rewriting there can produce an identical-looking 404 that has nothing to do with your application code.

Size it properly next time

Free calculators that prevent this class of failure before you provision hardware.

Free Tool

On-Prem AI Deployment Checklist

A 30-point pre-deployment checklist covering use cases, hardware, security, model operations, and rollout for self-hosted enterprise LLMs.

Free Tool

LLM Serving Capacity Planner

Convert a peak concurrent user target directly into a required GPU count with redundancy, then see the daily token and response capacity that hardware delivers.

Related problems

vLLM server won't start (port in use, auth, VRAM, or unsupported architecture)

vLLM server startup failures collapse into four buckets: the port is already bound by another process, Hugging Face auth is missing or expired for a gated repo, there isn't enough free VRAM for the requested model and context, or the installed vLLM version doesn't yet support the model's architecture. Read the last traceback line, not just the top, to tell them apart.

vLLM error: no chat template found for this model

The /v1/chat/completions endpoint needs a Jinja chat template to turn a messages array into the model's expected prompt format, and base pretrained checkpoints plus some older fine-tunes simply don't ship one. Fix it by supplying --chat-template pointing at a template file matching the model family, or by switching to the raw /v1/completions endpoint with a manually formatted prompt.

vLLM ignores tool or function calls, or returns them as plain text

Unlike the hosted OpenAI API, vLLM does not enable tool or function calling by default. You must launch with --enable-auto-tool-choice plus a --tool-call-parser matching your specific model family, and the model itself must have been trained to emit tool-call syntax its parser recognizes. Without both pieces, requests either error out or the model just writes the function call as plain text in its response content.

Guide

vLLM Production Deployment: A Practitioner's Guide

Deploy vLLM in production: continuous batching, PagedAttention, config flags that matter, and the metrics to watch before you trust it with real traffic.

Guide

LLM Observability: TTFT, ITL, Throughput, and GPU Dashboards

LLM inference observability: track TTFT, inter-token latency, throughput, and GPU utilization with dashboards that catch problems before users report them.

Still stuck, or tired of fighting your own infrastructure?

Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.

Talk to an On-Prem AI EngineerBrowse All Fixes
Back to Troubleshooting