A Low-Latency Architecture for Voice Agents with Live Web Retrieval

Cerebrium + Linkup + LiveKit

The hidden tradeoff in production voice agents

Building a real-time voice agent forces teams into a difficult constraint: latency vs. intelligence.

Unlike text chat, voice interactions operate within a strict latency budget. If responses take too long, the conversation breaks down. Production systems generally target sub-second time-to-first-audio from the end of a user's utterance — beyond ~1.5–2s of silence, the interaction feels broken.

A typical voice pipeline looks like this:

User Speech → STT → LLM Reasoning → TTS → Audio Response

Each stage adds latency:

This leaves little headroom. To stay in budget, teams often reach for small, fast models (Gemini Flash, GPT-4.1-mini, Llama 3.1 8B, Ultravox). They respond quickly, but bring real limitations:

• Outdated knowledge (frozen training cutoff)

• Weaker reasoning

• Higher hallucination rates on factual queries

For anything beyond narrow use cases, teams face a subtle tradeoff: tolerate hallucinations, or accept the slower responses of larger models.

Two levers, not one: fast MoE inference and Web Search retreieval

This architecture attacks the tradeoff from both sides.

1. A fast MoE model instead of a small dense model. We serve Qwen/Qwen3.6-35B-A3B — a 35B-parameter Mixture-of-Experts model with only ~3B active parameters per token. You get large-model quality at small-model latency, especially with DFLASH speculative decoding on top. In production we measure 60–115 tokens/sec and 300–510 ms TTFT on a single NVIDIA B200.

2. Conditional retrieval for freshness. No model, however large, knows today's stock price. When a query needs live facts, the model calls a search API, retrieves grounded context, and answers from it. The catch: most search APIs add 800–1500 ms. Linkup's fast endpoint is optimized for AI web search retrieval and returns in sub-second time (~0.7 s measured) — and we still hide it behind a spoken filler (below).

Architecture: Cerebrium + Linkup + LiveKit

User Speech
  ↓
STT (Deepgram nova-3)
  ↓
LLM tool decision (Qwen3.6-35B-A3B on Cerebrium B200, via SGLang)
  ↓
Conditional tool call ──► Linkup Web Search API ──► Retrieved context
  ↓
LLM grounded response
  ↓
TTS (Cartesia sonic-2)
  ↓
Audio Response

The key design decision is conditional retrieval: the LLM only calls Linkup when a query needs external knowledge (current events, prices, company data, docs). For conversational turns it answers immediately, with no search.

Measured latency (live console session)

The honest takeaway: retrieval still costs real time (~0.7s), but Linkup's fast depth keeps it sub-second. The system stays conversational by speaking a filler the moment a search starts, so the user hears a response at ~1s while the search completes in the background.

The latency-masking technique

When the model calls search_web, the agent immediately says a short filler ("I'm checking the web for you right now") before awaiting Linkup. This is the single most important UX trick in the system — it converts the retrieval gap into a natural, responsive-feeling turn.

We use outputType: "searchResults" rather than "sourcedAnswer": Linkup returns raw ranked snippets and skips its own answer-composition step, which trims latency (and cost) since our LLM already synthesizes the spoken reply from the retrieved context.

@function_tool()
async def search_web(self, ctx: RunContext, query: str) -> str:
    """Search the web for real-time information using Linkup fast search."""
    ctx.session.say("I'm checking the web for you right now.",
                    allow_interruptions=True)
    data = await self._linkup_request("search", {
        "q": query,
        "depth": "fast",
        "outputType": "searchResults",  # raw snippets; our LLM composes the reply
        "maxResults": 3,
        "includeImages": False,
    })
    return _format_sourced_answer(data) if data else "NO_RESULTS: ..."

Implementing the pipeline

1. Deploy the LLM on Cerebrium

Qwen3.6-35B-A3B is served with SGLang + DFLASH speculative decoding on a B200, using a custom runtime. Key settings from sglang-llm/cerebrium.toml:

[cerebrium.deployment]
docker_base_image_url = "modalresearch/sglang:nightly-dev-cu13-20260619-patched"

[cerebrium.hardware]
compute = "BLACKWELL_B200"
gpu_count = 1

[cerebrium.scaling]
min_replicas = 1          # avoid cold starts on a large MoE
replica_concurrency = 16

[cerebrium.runtime.custom]
port = 8000
healthcheck_endpoint = "/health"
entrypoint = ["python3", "main.py"]

Core SGLang server args:

--speculative-algorithm DFLASH
--speculative-draft-model-path z-lab/Qwen3.6-35B-A3B-DFlash
--speculative-num-draft-tokens 16
--attention-backend trtllm_mha
--speculative-draft-attention-backend fa4
--mem-fraction-static 0.75
--reasoning-parser qwen3
--tool-call-parser qwen3_coder

Note on constrained decoding: DFLASH speculative decoding does not yet support grammar / JSON-schema constrained output. If you need structured JSON, disable DFLASH for those requests or post-process plain text.

2. Point the voice agent at your endpoint

The LiveKit agent uses an OpenAI-compatible client against the Cerebrium endpoint. Disabling "thinking" tokens is a real latency lever:

openai.LLM(
    model="Qwen/Qwen3.6-35B-A3B",
    base_url=os.getenv("LLM_BASE_URL"),   # <https://api.cerebrium.ai/v4/><project>/sglang-llm/v1
    api_key=os.getenv("CEREBRIUM_API_KEY"),
    temperature=0.4,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)

3. Register Linkup as a tool

The LLM is given a search_web tool (via LiveKit's @function_tool); when it detects a factual query it emits a tool call, Linkup returns ranked search results, and the snippets are injected into context for the final response.

Before / after

The result: fast and accurate voice agents

Voice latency constraints do not require sacrificing intelligence. By pairing a fast MoE model (Qwen3.6 35B-A3B + DFLASH on Cerebrium B200) with low-latency retrieval (Linkup) and a filler-masked conditional tool call, voice agents stay inside the conversational window while grounding answers in the live web — with transparent, reproducible latency numbers rather than best-case estimates.