This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Operations

Deployment, observability and troubleshooting.

Running it, and finding out why it is not running.

1 - Deployment

The image, the ports, and the order things must be deployed in.

The image

It is a heavy image and has to be: it owns the audio. RNNoise is compiled from vendored C, TEN VAD is a prebuilt .so, and libopus, libsoxr and libspeexdsp are linked for the transport and the DSP chain. ONNX Runtime is included for the turn and voice-activity models that cascade calls use, and ONNXRUNTIME_LIB points at it.

The runtime stage is Debian slim and carries only the binary plus those shared libraries. It owns no business data — no database, no agent store, no credentials.

Ports

PortWhatExpose it?
50052gRPC (agentsession.v1)Loopback or private network only.
8000/healthAs needed.
6060pprofBinds container-loopback; publishing it does nothing.

Health

GET /health returns ok. It is a liveness check, not a readiness one — the process is healthy with zero calls running, which is the normal state.

Deployment order

The engine holds no project and no credentials, so the client that sends them must be deployed first. Deploying the server ahead of the client means every call is refused for want of a key.

The order:

  1. Deploy the client with its placement configured.
  2. Confirm [AIRoute] … creds=supplied (N bytes) on a call.
  3. Deploy the server.

Reversing it produces a working-looking server and a fleet of refused calls.

Sizing

One process handles many concurrent calls; the limit is CPU, and the audio chain is what spends it. Each call runs denoise, foreground isolation and neural VAD every 20 ms, and a cascade call runs its turn model on top.

GOMEMLIMIT is derived from the cgroup limit when unset — the boot log says what it picked:

[runtime] GOMEMLIMIT=13589MiB (/proc/meminfo limit=15987MiB, 85%)

Rolling restarts

Stop drains in-flight RPCs and then escalates to a hard stop after a timeout. One long-lived session stream must not hold a deploy open indefinitely — a session stream ends when its call does, which can be minutes away.

Calls in flight during a restart end. There is no session migration.

2 - Observability

The log lines worth knowing, and what they tell you.

Everything goes to stdout, so docker logs is the whole interface. Every line for a call carries its bridge id, which makes one call one filter:

docker logs agent_server --tail 100
docker logs agent_server 2>&1 | grep 48c7ad0bc30f     # one call
docker logs -f agent_server                            # follow

There are no metrics endpoints. Ship stdout to wherever you keep logs.

At boot

The whole configuration is printed, including variables that are set but not read — usually a rename, or a setting that now travels with the call:

[env] ---- agent server configuration ----
[env]   APP_GRPC_LISTEN_PORT             50052   # gRPC port, all interfaces
[env]   ULAI_GRPC_API_KEY                set len=64 sha256:57a4cbf1   # demanded of callers; unset = NO AUTH
[env]   ONNXRUNTIME_LIB                  /usr/local/lib/libonnxruntime.so (file present)   # Smart Turn and Silero; the image sets it
[env]   NOT READ BY THIS PROCESS: AI_BACKEND APP_ENV APP_NAME SERVICE_HTTP_PORT
[env]   (set in the environment but nothing here looks at them — usually a rename)
[env] ---- end agent server configuration ----

Secrets are shown as a length and a hash, never a value. A path that names a missing file says !! NO SUCH FILE.

Also at boot:

[grpc] agentsession.v1 listening on [::]:50052 (AgentBridge served; AgentSession/AgentDispatch served)

Per call

LineMeans
[bridge] join: bridge=… agent_id=… participant=…The agent has a seat.
[AIRoute] ai_project=… creds=supplied (N bytes) (from the agent profile)The profile carried a key. The one to check first.
using the profile supplied with the requestNo database lookup happened.
[gemini] connected, greeting deferredThe backend handshake succeeded.
ready — mode=… llm=…@… tts=…@… …A cascade call is up, with every setting it is running on.
greeting primedThe agent is about to speak.
roster: N other participant(s) in roomWho else is there.

[AIRoute] never prints a credential. It shows creds=supplied (2347 bytes) or creds=engine default — the latter meaning the profile carried none, which is a refusal.

Audio

When a call’s profile sets audio_nothing, the chain is cut down to the denoiser and the call’s log says so loudly:

[AUDIO] audio_nothing — pure RNNoise (preprocessor + foreground gate disabled)

That line in production is almost always a mistake — see troubleshooting.

Prompts

PROMPT_LOG is on unless set to off. Every text turn and tool result sent to the model is logged as an [→AI] line — the persona, the nudges, the closing instruction — which is the only record of what the model was actually told.

Turn it off if prompts or caller details should not reach your log store. Audio is never logged either way.

3 - Troubleshooting

Symptoms, and what actually causes them.

Start here

docker ps --filter name=agent_server           # is it running?
curl -s http://localhost:8000/health           # → ok
docker logs agent_server --tail 100            # what happened

Every line for a call carries its bridge id, so one call is one filter:

docker logs agent_server 2>&1 | grep 48c7ad0bc30f

The call joins, then dies immediately

Look for backend connect failed: right after joined session=….

this call carried no credentials for project "…"

The profile carried a project but no key. The engine holds none of its own, so there is nothing to fall back onto.

Confirm it in this server’s log — creds=engine default on the [AIRoute] line means the call carried no key:

docker logs agent_server 2>&1 | grep AIRoute | tail -5

The fix is on the agent client, which is the thing that must send a key. Nothing you change on this host will help: it holds no credentials by design.

this call carried no AI project

Same shape, other field. The profile named neither a project nor a key.

insufficient authentication scopes

Seen on older builds that fell back to ambient credentials. The host’s own identity — a GCE instance service account — authenticated successfully and then failed the Live handshake, because it lacks the cloud-platform scope Vertex needs.

Current images refuse the call before this can happen, with a message naming the missing field. If you see this, you are on an older image — check the tag with docker inspect --format '{{.Config.Image}}' agent_server.

DetectAnsweringMachine needs a Classifier

A call asked for answering-machine detection on an image that could not supply a classifier for it. Current images turn detection off rather than failing the call. Upgrade the image, or have the client stop asking for AMD.

The agent joins but never speaks

Check for greeting primed. If it is absent, the backend never connected — see above. If it is present and there is still silence, the caller’s side is the place to look: the room roster line says whether anyone else is actually there.

A setting on this host has no effect

You set AI_BACKEND, AMD_ENABLED, AUDIO_NOTHING or similar here, and calls behave as if you had not. Those are no longer read from the environment — see no longer read.

ONNXRUNTIME_LIB … !! NO SUCH FILE at boot

The image sets this to its own copy of ONNX Runtime, so this means it was overridden. Cascade calls still run, but without the turn and voice-activity models: they fall back to the built-in VAD and end turns on silence alone. Remove the override.

Calls are refused with unauthorized

The caller’s key does not match ULAI_GRPC_API_KEY. The boot log shows a hash of what the server expects:

[env]   ULAI_GRPC_API_KEY   set len=64 sha256:57a4cbf1

If it says (unset), the server is accepting everything, and the failure is elsewhere.

A tool never runs

Tools are configured on the agent and answered by the agent client — there is nothing to enable on this host. A tool that never runs is a client-side problem.

A call that pauses, goes quiet and then apologises is usually a tool the client answered too slowly. See tools.

Captions stop after the first utterance

Every caller utterance landing in one caption is a segment that never closed. Fixed in current images; on an older one the caller’s transcript accumulates into a single bubble while the agent’s lines appear separately. Upgrade.

The caller sounds gated or clipped

Have the client send audio_nothing on one call. It strips the chain to the denoiser alone, so if the problem disappears it is the pre-clean stage or the near-field foreground gate. There is no host-wide switch for this any more.

It is a diagnostic, not a setting. The full chain is the tuned one, and the stage it removes is what keeps a noisy room out of the model.