Operations

• 1: Deployment
• 2: Observability
• 3: Troubleshooting

1 - Deployment

The image

It is a heavy image and has to be: it owns the audio. RNNoise is compiled from vendored C, TEN VAD is a prebuilt .so, and libopus, libsoxr and libspeexdsp are linked for the transport and the DSP chain.

The runtime stage is Debian slim and carries only the binary plus those shared libraries. It owns no business data — no database, no agent store, no credentials.

Ports

Health

GET /health returns ok. It is a liveness check, not a readiness one — the process is healthy with zero calls running, which is the normal state.

Deployment order

The engine holds no project and no credentials, so the client that sends them must be deployed first. Deploying the server ahead of the client means every call is refused for want of a key.

The order:

1. Deploy the client with its placement configured.
2. Confirm [AIRoute] … creds=supplied (N bytes) on a call.
3. Deploy the server.

Reversing it produces a working-looking server and a fleet of refused calls.

Sizing

One process handles many concurrent calls; the limit is CPU, and the audio chain is what spends it. Each call runs denoise, foreground isolation and neural VAD every 20 ms.

GOMEMLIMIT is derived from the cgroup limit when unset — the boot log says what it picked:

[runtime] GOMEMLIMIT=13589MiB (/proc/meminfo limit=15987MiB, 85%)

Rolling restarts

Stop drains in-flight RPCs and then escalates to a hard stop after a timeout. One long-lived session stream must not hold a deploy open indefinitely — a session stream ends when its call does, which can be minutes away.

Calls in flight during a restart end. There is no session migration.

2 - Observability

Everything goes to stdout, so docker logs is the whole interface. Every line for a call carries its bridge id, which makes one call one filter:

docker logs agent_server --tail 100
docker logs agent_server 2>&1 | grep 48c7ad0bc30f     # one call
docker logs -f agent_server                            # follow

There are no metrics endpoints. Ship stdout to wherever you keep logs.

At boot

The whole configuration is printed, including variables that are set but not read — usually a rename:

[env] ---- agent server configuration ----
[env]   APP_GRPC_LISTEN_PORT   50052   # gRPC port; host is 127.0.0.1
[env]   ULAI_GRPC_API_KEY      set len=64 sha256:57a4cbf1
[env]   GEMINI_PROJECT_ID      (unset)   # NOT READ — the client sends it per call
[env]   NOT READ BY THIS PROCESS: APP_ENV APP_NAME SERVICE_HTTP_PORT
[env]   (set in the environment but nothing here looks at them — usually a rename)
[env] ---- end agent server configuration ----

Secrets are shown as a length and a hash, never a value.

Also at boot:

aibackend: realtime = gemini-live (default)
aibackend: classify = gemini-live (default)
[grpc] agentsession.v1 listening on [::]:50052 (AgentBridge served; AgentSession/AgentDispatch served)

Per call

[AIRoute] never prints a credential. It shows creds=supplied (2347 bytes) or creds=engine default — the latter meaning the profile carried none, which is now a refusal.

Turn accounting

played=…ms is not duration=…ms. It is how much agent audio the caller actually heard, and it differs from wall-clock by every second of silence, ringing and caller speech. It is the number that reflects what was delivered.

The same distinction appears per turn: a turn-ended event carries how much of that turn was heard before it was cut.

Audio

With AUDIO_NOTHING=true set on this host, the chain is cut down to the denoiser and says so loudly:

[AUDIO] AUDIO_NOTHING=true — pure RNNoise (preprocessor + foreground gate disabled)

That line in production is almost always a mistake — see configuration.

Prompts

PROMPT_LOG=on logs the system instruction at connect. Useful exactly once, when an agent is behaving oddly and you want to see what it was actually told. Noisy forever after, and it puts the prompt in your log store.

3 - Troubleshooting

Start here

docker ps --filter name=agent_server           # is it running?
curl -s http://localhost:8000/health           # → ok
docker logs agent_server --tail 100            # what happened

Every line for a call carries its bridge id, so one call is one filter:

docker logs agent_server 2>&1 | grep 48c7ad0bc30f

The call joins, then dies immediately

Look for backend connect failed: right after joined session=….

this call carried no credentials for project "…"

The profile carried a project but no key. The engine holds none of its own, so there is nothing to fall back onto.

Confirm it in this server’s log — creds=engine default on the [AIRoute] line means the call carried no key:

docker logs agent_server 2>&1 | grep AIRoute | tail -5

The fix is on the agent client, which is the thing that must send a key. Nothing you change on this host will help: it holds no credentials by design.

this call carried no AI project

Same shape, other field. The profile named neither a project nor a key.

insufficient authentication scopes

Seen on older builds that fell back to ambient credentials. The host’s own identity — a GCE instance service account — authenticated successfully and then failed the Live handshake, because it lacks the cloud-platform scope Vertex needs.

Current images refuse the call before this can happen, with a message naming the missing field. If you see this, you are on an older image — check the tag with docker inspect --format '{{.Config.Image}}' agent_server.

DetectAnsweringMachine needs a Classifier

A call asked for answering-machine detection on an image that could not supply a classifier for it. Current images turn detection off rather than failing the call. Upgrade the image, or have the client stop asking for AMD.

The agent joins but never speaks

Check for greeting primed. If it is absent, the backend never connected — see above. If it is present and there is still silence, the caller’s side is the place to look: the room roster line says whether anyone else is actually there.

An unknown backend kills the process at boot

aibackend: AI_BACKEND="gemini-liv" is not a known backend

Deliberate. A whole deployment quietly running a vendor nobody asked for is worse than refusing to start. The same value in a request costs only that call.

Calls are refused with unauthorized

The caller’s key does not match ULAI_GRPC_API_KEY. The boot log shows a hash of what the server expects:

[env]   ULAI_GRPC_API_KEY   set len=64 sha256:57a4cbf1

If it says (unset), the server is accepting everything, and the failure is elsewhere.

A tool never runs

Tools are configured on the agent and answered by the agent client — there is nothing to enable on this host. A tool that never runs is a client-side problem.

A call that pauses, goes quiet and then apologises is usually a tool the client answered too slowly. See tools.

Captions stop after the first utterance

Every caller utterance landing in one caption is a segment that never closed. Fixed in current images; on an older one the caller’s transcript accumulates into a single bubble while the agent’s lines appear separately. Upgrade.

The caller sounds gated or clipped

Have the client send audio_nothing on one call, or set AUDIO_NOTHING=true on this host to test every call. It strips the chain to the denoiser alone, so if the problem disappears it is the pre-clean stage or the near-field foreground gate.

It is a diagnostic, not a setting. The full chain is the tuned one, and the stage it removes is what keeps a noisy room out of the model.