OpenAI-Compatible Endpoints for Open-Source LLMs

The fastest way to move off a hosted model without rewriting your application is to put your open-source model behind an OpenAI-compatible endpoint. You keep the OpenAI SDK, keep your prompts and tool-calling code, and change one line: the base_url. This guide covers what that compatibility actually buys you, what serverless GPU changes underneath it, the frameworks that expose the schema, and the cost math.

TL;DR: the drop-in base_url swap

An OpenAI-compatible server replicates OpenAI's /v1/chat/completions and /v1/embeddings routes, request/response schema, and auth conventions. Point your existing client at https://your-endpoint/v1, set the model name your server advertises, and existing code runs unchanged. As Spheron notes, vLLM's --served-model-name flag "creates an OpenAI-compatible endpoint your existing SDK code hits without changes."

What "OpenAI-compatible" means, and why it matters

OpenAI never formally standardized its API, but it became the de facto interface for LLMs. An OpenAI-compatible API is "any API that replicates common OpenAI interface, request/response schema, and authentication conventions." That matters for three reasons: it is a drop-in replacement for the hosted API, it enables seamless migration between providers or self-hosted deployments, and it gives consistent integration with the tools and frameworks already built around the OpenAI schema. In practice, "any modern open-source LLM can be served behind an OpenAI-compatible API." No lock-in, and every LangChain, LlamaIndex, or agent framework that speaks OpenAI speaks to your model too.

What serverless GPU changes

Self-hosting used to mean keeping a GPU box running 24/7. Serverless GPU changes the economics: workers scale to zero when idle and scale back up on demand, so you pay per use. RunPod, for example, lets you "set active workers when startup latency matters, or allow flex workers to scale down when traffic is intermittent" - the same tradeoff every serverless GPU platform manages between cost and cold-start latency.

The cost of scale-to-zero is the cold start. In our testing on Cerebrium, loading a TensorRT-LLM Llama 3 8B engine into GPU memory takes "roughly ~10-15s" on each cold start. Keeping a small pool of warm capacity hides that from users when latency is critical. For a deeper treatment of cold starts, see our write-up on serverless GPU cold starts.

Endpoints also come in two shapes. Queue-based endpoints suit asynchronous jobs, batch work, and retryable requests; load-balanced endpoints suit direct HTTP access, streaming, and custom FastAPI or Flask services. Chat completions for interactive apps generally want the load-balanced, streaming pattern.

Frameworks that expose the schema

vLLM ships an OpenAI-compatible server out of the box - the most common way to stand up a /v1 endpoint for a Hugging Face model. Ollama is the lightweight local option and also exposes an OpenAI-compatible surface, useful for development before you deploy. Our DeepSeek-R1 guide walks the full vLLM path end to end on serverless GPU.

Which framework wins depends on your metric. In our benchmarks of Llama 3.1 70B FP8 on a single H100 - 256 input tokens, batch size 1, measured as total roundtrip to the Cerebrium platform - vLLM delivered the lowest time-to-first-token at 123ms, while SGLang was the throughput winner at 460 tokens per second on a batch size of 64. Different frameworks, different sweet spots.

Performance techniques worth knowing

Three techniques do most of the heavy lifting behind these numbers:

• PagedAttention manages the KV cache in non-contiguous pages, cutting memory waste so you fit more concurrent sequences on one GPU.

• Continuous batching dynamically schedules requests so the GPU does not stall waiting for the slowest request in a fixed batch - Hugging Face describes it as "processing multiple conversations in parallel and swapping them out when they are done."

• KV cache reuse keeps already-computed attention state so decoding does not recompute the whole context each token.

Together they are why a single H100 can serve high concurrency at low latency. We push these further in running Llama 3 8B with TensorRT-LLM, where the tutorial achieves "~1700 output tokens per second (FP8) on a single Nvidia A10 instance."

Supported models and licensing

Because the endpoint is model-agnostic, your choice is driven by license and quality, not the API. Popular open-source families that serve cleanly behind an OpenAI-compatible endpoint include Llama, Mistral, Qwen, DeepSeek, Phi, and Gemma. Check each model's license before commercial use - permissive terms differ across families - and see our notes on serving fine-tuned LLMs when you own the checkpoint.

The cost argument

The reason teams do this is usually the bill. Self-hosting Llama 3.3 70B in FP8 on a single H100 works out to roughly $1.67 per million output tokens at $2.40/hr, versus $10 per million for GPT-4o output - a large gap that widens with volume. For a fuller model of throughput-driven economics, see LLM inference cost at scale, and for hardware tradeoffs, H100 vs H200 throughput.

How to choose

Start from your traffic shape. Spiky or low-volume workloads favor scale-to-zero and tolerate a cold start; latency-critical apps warrant warm capacity. Pick vLLM for lowest TTFT, SGLang for peak batch throughput, and validate on your own prompts - vendor specs rarely match your roundtrip numbers. The migration itself is cheap: swap the base_url, keep the SDK, and you own the model. When you outgrow a single provider, that same compatibility is what makes the next move painless. If you are comparing platforms, our serverless provider overview and Modal alternatives guide are good next reads.