GPT-Live-Like Multimodal Agent Fully On-Device with Gemma 4

OpenAI’s GPT-Live changed the bar for voice AI in July 2026.

The important change was that GPT-Live uses a full-duplex architecture that can listen and speak at the same time, make interaction decisions continuously, and delegate harder work to a frontier model without freezing the conversation.

If you are building similar products, it is easy to look at that architecture and conclude that a GPT-Live-class experience requires a proprietary end-to-end voice model.

But Parlor is an interesting counterexample.

It is a research-preview project that tries to reproduce many of the product behaviors of GPT-Live while running the interaction stack on a laptop.

The default path uses Gemma 4 through llama.cpp, a dedicated semantic turn detector, Kokoro TTS, browser-side voice activity detection, FastAPI, and WebSockets.

On an Apple M3 Pro, there is 0.7 seconds from end-of-utterance to first audio with its smallest Gemma 4 configuration.

More importantly, Parlor is useful source code for anyone building real-time agents because it expose the following:

A responsive agent is a coordinated real-time system with explicit control flow, state ownership, cancellation, scheduling, and latency budgets.

The architecture details

The current Parlor v2 pipeline looks roughly like this:

Code

text

Browser
microphone + camera
      |
      | WebSocket
      | PCM audio + JPEG frame
      v
FastAPI session runtime
      |
      +--> Smart Turn v3
      |      semantic end-of-turn decision
      |
      +--> Gemma 4 via llama.cpp
      |      audio + image + conversation context
      |      streamed text response
      |
      +--> Action head
      |      grammar-forced JSON
      |      timers / modes / research
      |
      +--> Kokoro TTS
      |      sentence-by-sentence audio
      |
      +--> Optional background reasoner
             OpenAI-compatible endpoint
             research continues asynchronously
      |
      | WebSocket
      | transcript + PCM chunks + state events
      v
Browser playback + transcript + controls

It is fully on-device, real-time multimodal AI, with features similar to GPT-Live.

OpenAI describes GPT-Live as genuinely full duplex: the voice model continuously processes input while producing output and can decide many times per second whether to speak, listen, pause, interrupt, or invoke a tool.

GPT-Live removes the external turn detector from the audio path.

Although Parlor still has turns, it overlaps much of the expensive work with the user’s speech, uses a small semantic model to decide when a turn is actually complete, streams TTS before generation finishes, and moves tools into a separate control path.

The result is a classic cascade architecture engineered hard enough to feel much less like a classic cascade.

Editor’s note: If you want to dive deep into Local LLMs and Agentic Stack, you can join our Agent Foundry program for hands-on, in-depth trainings.

Quick start: run it locally

Parlor v2 is a Python project using uv.

The current pyproject.toml constrains Python to the 3.12 series, and the repository requires a recent llama.cpp build because Gemma 4 audio support landed recently.

For the default model, plan for roughly 6 GB of free RAM.

The smaller e2b configuration fits in roughly 4 GB, while the 12b option needs around 8 GB.

The supported targets are Apple Silicon macOS and Linux with a supported GPU.

On macOS:

Code

bash

git clone https://github.com/fikrikarim/parlor.git
cd parlor

curl -LsSf https://astral.sh/uv/install.sh | sh
brew install llama.cpp

uv sync
uv run parlor

Then open:

http://localhost:8000

Grant microphone and camera permission and start talking.

The first run downloads the model assets automatically, about 5.7 GB for the default Gemma 4 E4B QAT model plus its multimodal projector, with TTS assets downloaded separately.

Important note: The repository enforces llama.cpp build b9503 or newer for Gemma 4 audio, and b9512 or newer for MODEL=12b. src/parlor/llama.py checks the installed binary and refuses to start if a known stale build is detected.

The project’s model selector is intentionally small:

The current configuration is environment-variable driven.

A minimal .env might look like this:

Code

text

MODEL=e4b
PORT=8000
LLAMA_CTX=16384
TEMPERATURE=0.7

If you do nothing else, Parlor stays local, background web research is opt-in.

Setting REASONER_API_KEY enables delegation to an OpenAI-compatible endpoint.

It defaults to OpenRouter, but the reasoner code is deliberately generic enough to point at another compatible service or even a second local llama-server.

Parlor’s default interaction path is on-device, enabling a remote reasoner means research tasks can leave the device.

First run the benchmark, then touch the code

The architectural experiments are treated as benchmarks here.

Before changing the pipeline, run the end-to-end latency benchmark:

Code

text

uv run parlor

In another terminal:

Code

text

uv run python benchmarks/bench.py \
--label before \
--out benchmarks/results/before.json

After your change:

Code

text

uv run python benchmarks/bench.py \
--label after \
--out benchmarks/results/after.json

uv run python benchmarks/compare.py \
benchmarks/results/before.json \
benchmarks/results/after.json

The test suite also launches the real stack rather than mocking everything away:

Code

text

uv run pytest

The suite covers roughly 87 end-to-end tests and includes synthesized speech with clipped endings, noise, and other voices to reproduce live-microphone failures.

Latency, interruption, silence, echo, and malformed audio are your production environment.

Gemma 4 is the multimodal core but it is not the whole assistant

Parlor runs Google’s Gemma 4 through llama.cpp using quantized GGUF weights.

Google’s official Gemma 4 model card states that the E2B, E4B, and 12B variants support text, image, and audio input.

Gemma 4 E2B and E4B are specifically positioned for efficient deployment, while the 12B model also supports audio and vision.

llama.cpp’s current multimodal documentation lists Gemma 4 among models supporting mixed image and audio input through libmtmd and its OpenAI-compatible chat-completions server.

Parlor wraps that server directly.

A simplified version of the repository’s startup path looks like this:

Code

text

_proc = subprocess.Popen(
  cmd + [
      "-m", model,
      "--mmproj", mmproj,
      "-ngl", "99",
      "--port", str(PORT),
      "-c", str(CTX),
      "-np", "1",
  ],
  stdout=subprocess.DEVNULL,
  stderr=subprocess.DEVNULL,
)

Then every model request is sent to the local /v1/chat/completions endpoint with prompt caching enabled:

Code

json

body = {
  "messages": messages,
  "max_tokens": max_tokens,
  "temperature": TEMPERATURE if temperature is None else temperature,
  "stream": stream,
  "cache_prompt": True,
  "chat_template_kwargs": {"enable_thinking": False},
}

The design decision is what Parlor does not ask Gemma to do. It does not ask

the main model to own timers.
the main speech response to contain tool tags.
Gemma to decide end-of-turn timing.

And it does not rely on the conversational model to wake itself up later.

Those responsibilities belong to specialized components and deterministic runtime code.

Turn-taking is a two-stage problem

A naive voice agent usually does something like this:

Code

text

speech detected
-> record
-> silence detected
-> send to model
-> answer

This feels bad surprisingly quickly.

Humans pause in the middle of thoughts.

A 200–300 ms silence can mean “I am done,” but it can also mean “I am choosing the next word.” If every pause commits a turn, the assistant constantly interrupts.

Parlor therefore separates speech segmentation from turn completeness.

The browser uses a Silero-based VAD implementation to detect speech activity, via ricky0123/vad, which runs Silero VAD in the browser through ONNX Runtime Web.

Once a candidate utterance ends, the server runs Pipecat’s Smart Turn v3, an 8M-parameter semantic turn model based on a Whisper Tiny encoder plus a small classifier.

The Parlor wrapper takes the last eight seconds of 16 kHz audio and returns a probability:

Code

json

max_samples = WINDOW_SECONDS * SAMPLE_RATE

if len(audio) > max_samples:
  audio = audio[-max_samples:]
elif len(audio) < max_samples:
  audio = np.pad(audio, (max_samples - len(audio), 0))

log_mel = compute_whisper_log_mel_features(audio, do_normalize=True)

outputs = self._session.run(
  None,
  {"input_features": np.expand_dims(log_mel, 0)},
)
probability = outputs[0][0].item()

return probability > 0.5, probability

The server then decides whether to answer or keep holding the audio:

Code

json

complete, prob = await asyncio.get_event_loop().run_in_executor(
  None,
  detector.predict,
  pcm,
)

if not complete and not interrupted.is_set():
  held_audio = audio_b64s
  await send_json(ws, {
      "type": "turn_incomplete",
      "decision_s": decision_s,
      "p_complete": p_complete,
  })
  await prime(held_audio)
  continue

This is a small architecture decision with a large UX impact.

The VAD answers: did speech stop?
Smart Turn answers: did the thought end?

Those are different questions and treating them as the same problem is why many voice agents feel eager and interruptive.

Parlor’s own turnbench results show Smart Turn at 0.96 accuracy and roughly 19 ms on its labeled test set, while asking the Gemma variants to make the same FINISHED/WAIT decision was dramatically slower and near chance.

That is a useful agentic engineering lesson: a smaller specialized model can be much better than ask the main LLM again.

The latency trick: prefill while the user is still talking

The most interesting performance optimization is hidden in the browser/server protocol.

Parlor does not wait until the user finishes talking before it starts feeding the multimodal context through Gemma.

When speech starts, the browser captures a camera frame and sends it immediately.

During a long utterance, it sends roughly three-second speech chunks to the server.

The frontend contains this logic:

Code

typescript

const CHUNK_SECONDS = 3.0;

function sendSpeechChunk(upTo) {
if (!wsOpen()) return;
const chunk = concatFrames(uttFrames)
  .subarray(uttSamplesSent, upTo);
if (chunk.length < 4800) return;
wsSend({
  type: 'speech_chunk',
  seq: chunkSeq++,
  audio: float32ToWavBase64(chunk),
});
uttSamplesSent = upTo;
}

It also sends the camera frame at speech start:

Code

typescript

function prefetchFrame() {
if (framePrefetched || !cameraEnabled || !wsOpen()) return;

const image = captureFrame();
  if (!image) return;
  wsSend({ type: 'frame', image });
  framePrefetched = true;
}

On the server, these partial inputs are pushed through llama.cpp with a throwaway one-token request:

Code

python

async def prime_cache(messages: list) -> None:
  try:
      await asyncio.get_event_loop().run_in_executor(
          None,
          lambda: llama.chat_blocking(messages, max_tokens=1),
      )
  except Exception as e:
      print(f"Cache priming failed: {e}")

The response is irrelevant, and the prefix cache is the product.

By the time the user finishes a nine-second question, much of the image and audio prefill has already happened.

The final request only needs to process the tail.

This is the kind of optimization developers often miss because they benchmark model tokens per second instead of user-perceived critical path.

A real-time agent has a major advantage over a batch chatbot: the user gives you compute time while they are speaking.

Parlor spends it.

Camera input is not a video stream, a snapshot

The UI may feel multimodal and continuous, but the implementation is deliberately cheaper.

Parlor captures a downscaled JPEG frame from the browser camera:

Code

typescript

function captureFrame() {
if (!cameraEnabled || !video.videoWidth) return null;

const canvas = document.createElement('canvas');
  const scale = 320 / video.videoWidth;
  canvas.width = 320;
  canvas.height = video.videoHeight * scale;
  canvas.getContext('2d')
    .drawImage(video, 0, 0, canvas.width, canvas.height);
  return canvas
    .toDataURL('image/jpeg', 0.7)
    .split(',')[1];
}

That means camera support here should not be confused with continuously streaming video understanding.

It is a per-turn visual snapshot, which is a sensible engineering tradeoff for laptop inference.

For many agent use cases, this is enough: “what am I holding?”, “read this label,” “look at this error message,” or “what do you see?” do not need 30 frames per second.

Transcript-first generation is doing two jobs

Parlor asks Gemma to begin every audio response with a transcript marker:

Code

text

###TRANSCRIPT: <what the user said>
<assistant response>

At first glance that looks like formatting plumbing but it is actually a latency and reliability mechanism.

The transcript arrives first, so the browser can show what the model heard while the rest of the answer is still decoding.

More importantly, the repository reports that forcing the model to commit to the transcription before answering improved transcription accuracy in its internal test: leading transcript generation scored WER 0.00 versus 0.39 when the transcript came after the response on a long clean utterance.

The streaming parser then emits complete sentences to TTS as soon as they exist:

Code

text

parser = StreamParser(expect_transcript)

... await dispatch(parser.feed(item))

Code

python

...
async def dispatch(sentences):
  for sentence in sentences:
      await send_json(
          ws,
          {"type": "text_delta", "text": sentence + " "},
      )
      sentence_q.put_nowait(sentence)

The TTS worker consumes that queue in parallel with ongoing LLM decoding.

This produces an important pipeline shape:

The assistant does not wait for the complete textual answer before it begins speaking.

Do not put agent actions inside the speech stream

Many LLM agents mix natural language and control markup:

Sure, I'll set that timer. <tool name="timer" seconds="180" />

Parlor originally used that kind of in-band control.

The problem is obvious once you build a voice product: the user may hear “I’ll set a timer,” while the tool tag is malformed, omitted, truncated, or accidentally spoken aloud.

The assistant has made a promise the runtime did not execute.

Parlor v2 moves action detection into a separate grammar-forced JSON request over the same cached conversation.

The schema is small:

Code

json

HEAD_SCHEMA = {
  "type": "object",
  "properties": {
      "timer_seconds": {"type": "integer"},
      "timer_label": {"type": "string"},
      "mode": {
          "type": "string",
          "enum": [
              "none",
              "translate",
              "listen",
              "conversation",
          ],
      },
      "research_task": {"type": "string"},
  },
  "required": [
      "timer_seconds",
      "timer_label",
      "mode",
      "research_task",
  ],
}

The head runs deterministically:

Code

text

raw = llama.chat_blocking(
  build(head_prompt),
  max_tokens=192,
  temperature=0.0,
  json_schema=HEAD_SCHEMA,
)

Because llama.cpp compiles the JSON schema to a grammar, the control path gets structured output while the speech path remains pure natural language.

Treat conversation as the data plane and agent actions as the control plane.

The model can participate in both, but they should not be the same output stream.

The same model can have two personalities

There is another subtle optimization here.

Parlor does not load a second local model for action classification.

The action head runs on the same llama-server as the conversational response.

Why?

Prefix reuse.

After the speech response, the model server already has the conversation, multimodal input, and generated confirmation in cache.

The action request appends a small instruction and decodes roughly a few dozen JSON tokens.

So Parlor gets a kind of “multi-head agent” behavior without a multi-model memory penalty:

Code

text

same Gemma 4 context
      |
      +--> sampled conversational speech
      |
      +--> temperature-0 structured action decision

Background research is asynchronous by design

GPT-Live’s public architecture separates continuous interaction from deeper work.

Parlor imitates that behavior with an optional background reasoner.

When enabled, the action head can emit a research_task.

The server starts the work as an asyncio task using a dedicated thread pool, while the conversational session continues.

The reasoner itself is intentionally boring: an OpenAI-compatible /chat/completions request with a system prompt asking for a short, spoken-style answer.

The interesting code is the orchestration around it.

The completed task is pushed back into the same message queue used by the real-time session.

It is not spoken immediately if the conversational “floor” is busy.

Parlor explicitly tracks whether the user is speaking, audio is being held, a barge-in is active, or assistant playback is still running.

Finished research results and timer events are serialized around those states.

Conceptually:

Code

text

research completes
    |
    v
ready_events queue
    |
    +--> user speaking? hold
    |
    +--> assistant speaking? hold
    |
    +--> translation/listen mode? maybe hold
    |
    +--> floor free? deliver

This is agent concurrency control, and without it, an asynchronous agent becomes a race condition with a voice.

Barge-in is cancellation

When the user starts speaking over the assistant, the browser stops playback and sends an interrupt event.

The server does not merely ignore the rest of the text.

ChatStream.cancel() closes the socket used for the llama.cpp generation request so inference is actually aborted server-side.

The frontend also uses a sustained-speech gate rather than reacting to one VAD spike, because the assistant’s own TTS can leak back into the microphone.

That code is a reminder that real-time voice systems have to distinguish three different operations:

stop playing audio stop generating tokens start capturing the user's new utterance

Parlor treats barge-in as a cross-layer state transition.

The browser is part of the agent runtime

It is tempting to think of the frontend as a microphone button and a transcript view but Parlor’s browser code is doing much more.

It owns speech activity frames, pre-speech buffering, barge-in detection, camera capture, audio chunking, streamed playback scheduling, a flush timer for falsely held incomplete turns, visible timer state, research-task chips, translation/listen mode controls, and a watchdog that prevents the UI from getting stuck in “processing.”

Parlor includes manual escape hatches for stateful modes.

If the user says “stop translating” and the model mishears it, the mode chip has a stop button.

Timer chips have cancellation controls. A reconnect resets state because the server-side conversation is gone.

That is a good instinct:

Never make the model the only way to escape a model-created state.

Concluding thoughts

The reusable idea here is that the agent should be designed like a distributed system even when every component runs on one laptop.

There are:

concurrent actors
messages
queues
cancellations
authoritative state owners
speculative operations
fallbacks
user-visible consistency guarantees

Parlor is still rough, local, and experimental but as a piece of engineering source code, it shows where the hard work is moving in voice agents.