Running it, and finding out why it is not running.
This is the multi-page printable view of this section. Click here to print.
Operations
1 - Deployment
The image
It is a heavy image and has to be: it owns the audio. RNNoise is compiled
from vendored C, TEN VAD is a prebuilt .so, and libopus, libsoxr and
libspeexdsp are linked for the transport and the DSP chain. ONNX Runtime is
included for the turn and voice-activity models that cascade calls use, and
ONNXRUNTIME_LIB points at it.
The runtime stage is Debian slim and carries only the binary plus those shared libraries. It owns no business data — no database, no agent store, no credentials.
Ports
| Port | What | Expose it? |
|---|---|---|
50052 | gRPC (agentsession.v1) | Loopback or private network only. |
8000 | /health | As needed. |
6060 | pprof | Binds container-loopback; publishing it does nothing. |
The gRPC server has no transport credentials of its own. The API key and any service-account key a client sends cross the wire in clear.
Publish 50052 to 127.0.0.1 and terminate TLS in front of it. The SDK’s
Secure() expects exactly that.
Health
GET /health returns ok. It is a liveness check, not a readiness one — the
process is healthy with zero calls running, which is the normal state.
Deployment order
The engine holds no project and no credentials, so the client that sends them must be deployed first. Deploying the server ahead of the client means every call is refused for want of a key.
The order:
- Deploy the client with its placement configured.
- Confirm
[AIRoute] … creds=supplied (N bytes)on a call. - Deploy the server.
Reversing it produces a working-looking server and a fleet of refused calls.
Sizing
One process handles many concurrent calls; the limit is CPU, and the audio chain is what spends it. Each call runs denoise, foreground isolation and neural VAD every 20 ms, and a cascade call runs its turn model on top.
GOMEMLIMIT is derived from the cgroup limit when unset — the boot log says
what it picked:
[runtime] GOMEMLIMIT=13589MiB (/proc/meminfo limit=15987MiB, 85%)
Rolling restarts
Stop drains in-flight RPCs and then escalates to a hard stop after a timeout.
One long-lived session stream must not hold a deploy open indefinitely — a
session stream ends when its call does, which can be minutes away.
Calls in flight during a restart end. There is no session migration.
2 - Observability
Everything goes to stdout, so docker logs is the whole interface. Every line
for a call carries its bridge id, which makes one call one filter:
docker logs agent_server --tail 100
docker logs agent_server 2>&1 | grep 48c7ad0bc30f # one call
docker logs -f agent_server # follow
There are no metrics endpoints. Ship stdout to wherever you keep logs.
At boot
The whole configuration is printed, including variables that are set but not read — usually a rename, or a setting that now travels with the call:
[env] ---- agent server configuration ----
[env] APP_GRPC_LISTEN_PORT 50052 # gRPC port, all interfaces
[env] ULAI_GRPC_API_KEY set len=64 sha256:57a4cbf1 # demanded of callers; unset = NO AUTH
[env] ONNXRUNTIME_LIB /usr/local/lib/libonnxruntime.so (file present) # Smart Turn and Silero; the image sets it
[env] NOT READ BY THIS PROCESS: AI_BACKEND APP_ENV APP_NAME SERVICE_HTTP_PORT
[env] (set in the environment but nothing here looks at them — usually a rename)
[env] ---- end agent server configuration ----
Secrets are shown as a length and a hash, never a value. A path that names a
missing file says !! NO SUCH FILE.
Also at boot:
[grpc] agentsession.v1 listening on [::]:50052 (AgentBridge served; AgentSession/AgentDispatch served)
Per call
| Line | Means |
|---|---|
[bridge] join: bridge=… agent_id=… participant=… | The agent has a seat. |
[AIRoute] ai_project=… creds=supplied (N bytes) (from the agent profile) | The profile carried a key. The one to check first. |
using the profile supplied with the request | No database lookup happened. |
[gemini] connected, greeting deferred | The backend handshake succeeded. |
ready — mode=… llm=…@… tts=…@… … | A cascade call is up, with every setting it is running on. |
greeting primed | The agent is about to speak. |
roster: N other participant(s) in room | Who else is there. |
[AIRoute] never prints a credential. It shows creds=supplied (2347 bytes)
or creds=engine default — the latter meaning the profile carried none, which
is a refusal.
Audio
When a call’s profile sets audio_nothing, the chain is cut down to the
denoiser and the call’s log says so loudly:
[AUDIO] audio_nothing — pure RNNoise (preprocessor + foreground gate disabled)
That line in production is almost always a mistake — see troubleshooting.
Prompts
PROMPT_LOG is on unless set to off. Every text turn and tool result sent to
the model is logged as an [→AI] line — the persona, the nudges, the closing
instruction — which is the only record of what the model was actually told.
Turn it off if prompts or caller details should not reach your log store. Audio is never logged either way.
3 - Troubleshooting
Start here
docker ps --filter name=agent_server # is it running?
curl -s http://localhost:8000/health # → ok
docker logs agent_server --tail 100 # what happened
Every line for a call carries its bridge id, so one call is one filter:
docker logs agent_server 2>&1 | grep 48c7ad0bc30f
The call joins, then dies immediately
Look for backend connect failed: right after joined session=….
this call carried no credentials for project "…"
The profile carried a project but no key. The engine holds none of its own, so there is nothing to fall back onto.
Confirm it in this server’s log — creds=engine default on the [AIRoute]
line means the call carried no key:
docker logs agent_server 2>&1 | grep AIRoute | tail -5
The fix is on the agent client, which is the thing that must send a key. Nothing you change on this host will help: it holds no credentials by design.
this call carried no AI project
Same shape, other field. The profile named neither a project nor a key.
insufficient authentication scopes
Seen on older builds that fell back to ambient credentials. The host’s own
identity — a GCE instance service account — authenticated successfully and then
failed the Live handshake, because it lacks the cloud-platform scope Vertex
needs.
Current images refuse the call before this can happen, with a message naming
the missing field. If you see this, you are on an older image — check the tag
with docker inspect --format '{{.Config.Image}}' agent_server.
DetectAnsweringMachine needs a Classifier
A call asked for answering-machine detection on an image that could not supply a classifier for it. Current images turn detection off rather than failing the call. Upgrade the image, or have the client stop asking for AMD.
The agent joins but never speaks
Check for greeting primed. If it is absent, the backend never connected —
see above. If it is present and there is still silence, the caller’s side is
the place to look: the room roster line says whether anyone else is actually
there.
A setting on this host has no effect
You set AI_BACKEND, AMD_ENABLED, AUDIO_NOTHING or similar here, and calls
behave as if you had not. Those are no longer read from the environment — see
no longer read.
ONNXRUNTIME_LIB … !! NO SUCH FILE at boot
The image sets this to its own copy of ONNX Runtime, so this means it was overridden. Cascade calls still run, but without the turn and voice-activity models: they fall back to the built-in VAD and end turns on silence alone. Remove the override.
Calls are refused with unauthorized
The caller’s key does not match ULAI_GRPC_API_KEY. The boot log shows a hash
of what the server expects:
[env] ULAI_GRPC_API_KEY set len=64 sha256:57a4cbf1
If it says (unset), the server is accepting everything, and the failure
is elsewhere.
A tool never runs
Tools are configured on the agent and answered by the agent client — there is nothing to enable on this host. A tool that never runs is a client-side problem.
A call that pauses, goes quiet and then apologises is usually a tool the client answered too slowly. See tools.
Captions stop after the first utterance
Every caller utterance landing in one caption is a segment that never closed. Fixed in current images; on an older one the caller’s transcript accumulates into a single bubble while the agent’s lines appear separately. Upgrade.
The caller sounds gated or clipped
Have the client send audio_nothing on one call. It strips the chain to the
denoiser alone, so if the problem disappears it is the pre-clean stage or the
near-field foreground gate. There is no host-wide switch for this any more.
It is a diagnostic, not a setting. The full chain is the tuned one, and the stage it removes is what keeps a noisy room out of the model.