Self-Hosted LLM Cost-Per-Request & Token Tracking

When you call OpenAI or Anthropic, the bill arrives itemized: every request, its token counts, and a dollar figure attached. The moment you move to a self-hosted open model on your own GPUs, that statement stops coming. Nobody sends you a per-request ledger anymore - you have to generate one yourself. This guide is for the Lead ML Engineer who already runs open-model inference and now needs to rebuild the token-usage visibility, cost-per-request attribution, and audit trail that vanished with the provider dashboard.

The provider-dashboard gap: why self-hosting erases usage visibility

A managed API abstracts a lot of accounting away from you. Usage metering, per-key attribution, and the cost math all happen on the provider's side, and you consume the result as a dashboard and an invoice. Self-hosting inverts that. You rent GPUs by the hour (or the second), and the hardware meter keeps ticking whether a request is in flight or the replica is idle. There is no line item that says what any single request cost you.

Worse, the dashboards you do get are often the wrong artifact for the job. As one practitioner running their own proxy puts it, a pretty visual is "not token usage statistics with time series data meant for engineers." Engineers making infrastructure decisions need queryable time-series data, not a summary tile. That gap - between what the GPU bill tells you (an hourly rate) and what you actually need to allocate cost (a per-request token count) - is the whole problem.

Operational observability is a benefit of ownership - if you instrument it

Here is the upside nobody mentions when they warn you off self-hosting: owning the infrastructure gives you more observability potential than any managed API ever will, because every request passes through hardware you control. The same proxy operator notes that when something goes wrong - "a cache miss during a coding session, I can pinpoint why that happened in 20 seconds by hand." You cannot do that against a black-box endpoint.

But observability is a potential, not a default. It only exists if you instrument it. The open model ships with the weights and nothing else - no metering, no attribution, no retention policy. The visibility is yours to build, and the cost of not building it is that you are flying blind on the single largest line in your infrastructure budget.

Per-request logging at the proxy layer: the data model

The cleanest place to capture everything is the proxy that sits in front of your inference endpoints. It is the natural chokepoint: every request has to pass through it, so it is the one point where you can observe, stamp, and record every invocation without touching model code.

At minimum, each log line should answer which key called what model with what client at what time, and its status - the fields a self-hosted proxy operator considers table stakes: "logs of which key called what model with what client at what time and its status." Concretely, the per-request record is:

• API key / caller identity - who or what made the call, so you can attribute cost to a team, service, or customer.

• Model (and version) - which model served the request.

• Client - the application or agent that originated it.

• Timestamp - for time-series rollups and rate analysis.

• Status - success, error, timeout, so failed spend is visible too.

• Input and output token counts - the raw units of cost.

That last field is the one that turns logs into a bill. Tokens are the unit; cost is tokens multiplied by your true cost-per-token. On self-hosted hardware that rate is derived from throughput: the faster your stack moves tokens through a rented GPU, the less each token costs. In our testing, a TensorRT-LLM Llama 3 8B build reaches ~1700 output tokens per second (FP8) on a single Nvidia A10 instance, and our benchmarks of serving frameworks found SGLang hitting 460 tokens per second on a batch size of 64 for Llama 3.1 70B FP8 on one H100. Divide the hourly GPU rate by tokens-per-hour at your real batch size and you have the per-token price that makes each logged request a line item. The full tokens-per-minute cost model is worked through in our LLM inference cost at scale guide, and the framework throughput numbers behind the per-token math live in our vLLM, SGLang and TensorRT benchmark.

The governance and audit-trail deficit of open models

Cost attribution is half the story. The other half is governance. Open models running on local runtimes ship with an audit-trail deficit: as the Hugging Face community frames it, "when agents run on open models via local runtimes, governance often gets ignored: who ran what, on which model, and at what token cost." There is no provider-side record of who did what - if you do not log it, it did not happen, as far as any future audit is concerned.

The same proxy log that produces your cost line items is also your audit trail. The two deficits - no bill, no audit record - have a single fix.

Per-run accountability: model version, source, user, tokens

To make the audit trail defensible, each run needs more than a token count. The governance-minded version of the record is that "every agent run logs its model, trigger source, responsible person and token usage." Mapped to fields:

• Model version - not just "Llama 3" but the exact artifact, so you can tie behavior to a deployment. Our note on managing model versions across staging and production covers keeping that identifier stable.

• Trigger source - the invocation origin (which agent, cron, or endpoint).

• Responsible person - the human or team accountable for the run.

• Token consumption - the per-run usage that doubles as the cost unit.

If you serve an OpenAI-compatible endpoint for open-source LLMs, the request already carries most of these fields; the proxy's job is to persist them rather than discard them after the response streams back.

Putting it together: a reference approach

1. Put a proxy in front of every endpoint. Route all inference through it so there is one chokepoint and no un-instrumented path to the GPU.

2. Stamp every request. Capture key, model and version, client, trigger source, responsible person, timestamp, status, and input/output tokens.

3. Persist to a time-series store. Keep the raw rows queryable - engineers need the series, not a summary tile.

4. Attach a cost-per-token derived from measured throughput. Multiply logged tokens by your real rate so each row becomes a dollar figure.

5. Roll up by key, team, and model for the itemized statement the provider used to send you.

Done well, this is not overhead - it is leverage. The same instrumentation that gives you an audit trail also surfaces where spend concentrates, which is exactly how teams cut it: DistilLabs, running single-digit-millions of requests a day, worked with us to reduce inference costs by 50% while raising accuracy from 83% to 92%. You cannot optimize a bill you cannot see. For the economics beneath the per-token rate, see our LLM inference cost at scale and serving fine-tuned LLMs guides, and if cold-start idle time is distorting your cost-per-request, our autoscaling Llama 70B write-up explains the replica-startup lever.

Frequently asked questions

Why does self-hosting an open model remove per-request cost visibility?

Because there is no provider in the loop to meter and invoice you. You rent GPUs by time, and the hardware bill reflects hours, not requests. The only way to recover a per-request cost line item is to log token counts yourself and multiply them by a cost-per-token you derive from your own throughput.

What fields should each per-request log line contain?

At minimum: which API key called which model with which client at what time and its status, plus input and output token counts. For an audit-grade trail, add model version, trigger source, and the responsible person so every run is traceable.

Where is the right place to capture this data?

At the proxy in front of your endpoints. Every request passes through it, so it is the single chokepoint where you can stamp and persist the full record without modifying model code.

How do I turn logged tokens into a dollar cost?

Tokens are the unit; cost is tokens times your true cost-per-token. Derive that rate from measured throughput on your hardware - for example, in our testing a TensorRT-LLM Llama 3 8B build reached ~1700 output tokens/sec (FP8) on a single A10 - then divide the GPU hourly rate by tokens-per-hour at your real batch size.