Agent Server

• 1: Overview
• 2: Deploy it
• 3: What a call carries
• 3.1: The agent profile
• 3.2: Tools
• 3.3: The shape of a call
• 4: Configuration
• 5: Interfaces
• 6: Operations
• 6.1: Deployment
• 6.2: Observability
• 6.3: Troubleshooting

docker run -d --name agent_server --restart unless-stopped \
  -p 127.0.0.1:50052:50052 \
  -p 8000:8000 \
  -e APP_GRPC_LISTEN_PORT=50052 \
  -e APP_HTTP_PORT=8000 \
  -e ULAI_GRPC_API_KEY='<64-character key>' \
  asia-south1-docker.pkg.dev/arctic-operand-415316/ulai/agent_server:v1.02

curl -s http://localhost:8000/health     # → ok

  agent client  ──gRPC :50052──▶  agent server  ──▶  Ulai SFU room
  (drives calls)                   (this image)       (the conversation)

1 - Overview

What it is

The Ulai Agent Server is a single Go binary (orchestrator) that runs one side of a voice conversation. It joins an Ulai SFU room as an ordinary participant, connects an AI backend to that room, and manages the conversation until somebody ends it.

Everything it owns is timing-critical:

• The audio chain — pre-clean, denoise, near-field foreground isolation, neural VAD.
• Turn detection — who has the floor, and when that changes.
• Barge-in — cutting the agent mid-sentence when the caller speaks, and accounting for how much of the turn was actually heard.
• The silence ladder — nudges when a caller goes quiet, and a hangup when they stay quiet.
• The end-call contract — the evidence a model must produce before it is allowed to hang up on a human.

What it is not

It is not a place where anything is stored. There is no database, no agent table, no recording bucket and no credential file. A call’s configuration arrives in the request that starts it, and everything worth keeping leaves on the event stream for somebody else to write down.

That is a deliberate constraint rather than an omission. It is what lets one engine serve agents belonging to different customers, billed to different GCP projects, running on different AI vendors — and it is what makes the binary safe to hand to somebody else.

Where it sits

  agent client  ──gRPC──▶  agent server  ──WebSocket──▶  Ulai SFU room
  (your logic)              (this)                        (the conversation)
       │                       │
       │                       └──▶ AI backend (Gemini Live, …)
       │
       └──▶ your database, your dashboards, your billing

The agent client decides which agent runs, what it says, and where it is billed. It sends all of that with each call. The agent server runs the call. The two speak over port 50052 — see interfaces.

A client that crashes mid-call does not drop the caller. The engine carries on with the configuration it was given; what is lost is the client’s ability to steer and to record, not the conversation.

How it is shipped

As a container image, and only as one. There is no source build, no package to install and no plugin to load — everything the engine needs is inside it, including the compiled audio libraries.

What you supply is a port, a key, and a client to drive it. See deploy it.

What it does not carry

Audio never leaves the media path. The agent client receives transcripts and sends text; no audio frame crosses the control port.

That is what keeps a slow or distant client from being heard: the round trip it adds lands between turns, where tens of milliseconds are invisible — not inside the frame cadence, where they are a stutter the caller hears.

2 - Deploy it

Before you start

You need:

• Docker, and access to the registry the image lives in.
• A key for port 50052 — any 64-character random string. The agent client must present the same one.
• An agent client to drive it. The agent server never places a call by itself; it waits to be told which room to join.

You do not need a GCP project, a service-account key, a model name or a prompt on this host. All of that arrives with each call.

1. Pull

docker login asia-south1-docker.pkg.dev
docker pull asia-south1-docker.pkg.dev/arctic-operand-415316/ulai/agent_server:v1.02

Pin the tag. latest makes it impossible to say afterwards what was running.

2. Run

docker run -d --name agent_server --restart unless-stopped \
  -p 127.0.0.1:50052:50052 \
  -p 8000:8000 \
  -e APP_GRPC_LISTEN_PORT=50052 \
  -e APP_HTTP_PORT=8000 \
  -e ULAI_GRPC_API_KEY='<64-character key>' \
  asia-south1-docker.pkg.dev/arctic-operand-415316/ulai/agent_server:v1.02

With docker compose

services:
  agent_server:
    image: asia-south1-docker.pkg.dev/arctic-operand-415316/ulai/agent_server:v1.02
    restart: unless-stopped
    ports:
      - "127.0.0.1:50052:50052"
      - "8000:8000"
    environment:
      APP_GRPC_LISTEN_PORT: 50052
      APP_HTTP_PORT: 8000
      ULAI_GRPC_API_KEY: "${ULAI_GRPC_API_KEY}"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 10s

3. Verify it started

curl -s http://localhost:8000/health

ok means the process is alive. It says nothing about calls — the server is healthy with zero calls running, which is its normal state.

Now read back what it thinks its configuration is:

docker logs agent_server 2>&1 | grep -A 25 "agent server configuration"

[env] ---- agent server configuration ----
[env]   APP_GRPC_LISTEN_PORT   50052   # gRPC port; host is 127.0.0.1
[env]   APP_HTTP_PORT          8000    # health only
[env]   ULAI_GRPC_API_KEY      set len=64 sha256:57a4cbf1
[env]   GEMINI_PROJECT_ID      (unset) # NOT READ — the client sends it per call
[env] ---- end agent server configuration ----
[grpc] agentsession.v1 listening on [::]:50052

Three things to confirm:

The log also prints any variable that is set but nothing reads — usually a rename, and worth a look.

4. Point a client at it

Configure the agent client with this host’s address and the same key. In a typical client that is:

Nothing appears in the agent server’s log until the client connects and starts a call. Silence here is normal.

5. Confirm a real call

Place one call through the client, then:

docker logs agent_server --tail 50

A healthy call looks like this:

[bridge] join: bridge=48c7ad… agent_id=d1a6b0… participant=agent_7f85…
[AIRoute] ai_project=my-project location=us-central1 creds=supplied (2347 bytes)
[bridge:48c7ad…] using the profile supplied with the request — no agent lookup, no database
[gemini] connected, greeting deferred (model=… voice=Kore)
[bridge:48c7ad…] greeting primed

Two lines are the ones to know:

• creds=supplied (N bytes) — the client sent credentials. If it says creds=engine default, the client sent none and the call will be refused.
• greeting primed — the backend connected and the agent is about to speak. If this never appears, the call died before it could talk.

Upgrading

docker pull …/agent_server:v1.03
docker stop agent_server && docker rm agent_server
docker run -d … …/agent_server:v1.03      # same flags

Calls in flight during a restart end — there is no session migration. Restart when the log is quiet, or accept the dropped calls.

Next

• Configuration — every variable.
• Troubleshooting — when a call does not work.

3 - What a call carries

The agent server is configured by two things, and only one of them is on this host.

3.1 - The agent profile

Almost nothing about a conversation is configured on the agent server. The prompt, the voice, the GCP project, the service-account key, which behaviours are on — all of it arrives with each call, in something called the agent profile, sent by the agent client.

This matters to you as an operator for three reasons.

1. It explains the short environment list

You are not missing settings. There is no PROMPT, no VOICE, no GEMINI_MODEL to set here, because those are properties of an agent, not of a host. One agent server runs many different agents at once, each with its own.

2. It means one host can serve many customers

Because the GCP project and the service-account key travel with each call, a single agent server can run:

• agent A, billed to customer A’s GCP project, on customer A’s key
• agent B, billed to customer B’s, on customer B’s key

at the same time. As host configuration this would be impossible — one deployment could only ever serve one account, and serving two meant running two servers.

3. It means this host holds no credentials

There is no service-account key on this machine and no file to mount. If somebody copies the image, they get no customer’s credentials with it.

The trade is that a call carrying no credentials cannot run. There is nothing to fall back onto, so it is refused:

gemini: this call carried no credentials for project "my-project" —
the agent server holds none of its own, by design. Set google_credentials_json
on the agent, or GOOGLE_APPLICATION_CREDENTIALS on the agent client that sends
its profile

If you see that, the fix is on the client, not here. See troubleshooting.

What is in a profile

You do not set these — the client does — but knowing what a call carries makes the logs readable.

Each field falls back on its own. A client sending a project but no region gets that project in the default region — not a half-configured call.

Where the environment still helps

A handful of environment variables act as fallbacks for calls whose profile leaves a field unset — the default AI backend, whether answering-machine detection runs. They are listed in configuration.

The project and the key are not among them. Those must come from the call.

3.2 - Tools

An agent can do more than talk. Mid-call it can look up a balance, an order status, an appointment slot — anything the business it belongs to can answer.

None of that work happens on this host. The agent server has no database and no idea what any of those tools mean. It carries the request out to the agent client and carries the answer back.

model asks for "check_balance"
        │
        ▼
  agent server ──▶ agent client ──▶ the business's own system
        ◀──────────── answer ◀───────────────┘
        │
        ▼
model continues the conversation

What this means operationally

Tools are configured on the agent, not here. There is nothing to install, declare or permit on this host to make a tool available.

A slow tool is heard as silence. While the client is answering, the model is not speaking, and the caller hears that. The wait is bounded and always ends in an answer to the model — if the client does not answer in time, the model is told the lookup failed so it can apologise and carry on. A caller hearing “I couldn’t pull that up just now” is a far better outcome than a line that goes quiet.

So a tool that is slow on the client side shows up as pauses in conversations, not as errors in this log.

Two tools belong to the server. end_call — how a model hangs up, subject to a contract that checks it is entitled to — and a no-op tool declared beside it. The second exists so that a model which feels the urge to “use a tool” at a moment that is not an ending has somewhere harmless to put it, rather than reaching for end_call and cutting a live human off.

It is answered instantly, inside the server, precisely because sending it out and back would spend a multi-second silence avoiding a mistake that costs nothing.

In the logs

Tool activity appears against the call’s bridge id. A call that pauses oddly, with the agent going quiet and then apologising, is usually a tool the client was slow to answer — the place to look is the client’s logs, not this one.

3.3 - The shape of a call

Joining

The room exists first. Whoever created it is the only party that knows which room the other participants are in, so the agent server never creates one — it is told which to join.

client ──JoinBridge──▶ server ──join──▶ control plane
                          │
                          ├──▶ SFU room     (WebSocket, Opus)
                          └──▶ AI backend   (the placement on the profile)

JoinBridge returns once the agent has a seat. The backend handshake continues after that.

Who speaks first

On a bridge the agent does. The room is already live, nobody rang, and an agent that joins and stays silent reads as a broken connection.

opening overrides this. Set it to caller when the agent is joining a conversation already in progress and should listen rather than announce itself.

During

The engine owns the timing. Every 20 ms of caller audio runs the chain — pre-clean, denoise, near-field foreground isolation, neural VAD — and the VAD is the authoritative turn driver.

The agent client receives transcripts and turn events and can steer between turns. What it cannot do is touch the media path: there is no way to send audio into the call and no way to change the audio processing. Timing stays with the engine.

Barge-in

When the caller speaks over the agent, the agent stops. What matters afterwards is not what the agent generated but what the caller actually heard, so a turn-ended event carries both:

An agent turn generated in full and heard for 300 ms did not happen, whatever the transcript says.

A client deciding “have they been told about the fee?” needs the second number, not the first.

Ending

Three ways a call ends:

end_call is not a request the model gets for free. It must produce evidence — which of the legitimate endings this is, whether the closing question was actually spoken and answered — and that evidence is cross-checked against what the engine independently observed. A rejected end_call is answered with a re-prompt, so the agent keeps talking rather than going silent.

Every rejection is a prompt, and the engine is careful that rejections do not themselves become a source of repetition.

Detaching is not hanging up

If the client process dies mid-call, the engine carries on. The caller is not dropped because the record-keeper crashed; the cost is a reporting gap, which is the right one to take.

4 - Configuration

Configuration is split by a single rule:

If it decides how a call runs, it travels with the call. If it decides how the process runs, it is an environment variable.

That split is why the environment list below is so short, and why the same binary can serve agents belonging to different customers at the same time. The per-call half is documented under the agent profile.

The process prints its whole configuration at boot, including which variables are set but not read — usually a rename:

[env] ---- agent server configuration ----
[env]   APP_GRPC_LISTEN_PORT             50052   # gRPC port; host is 127.0.0.1
[env]   ULAI_GRPC_API_KEY                set len=64 sha256:57a4cbf1
[env]   GEMINI_PROJECT_ID                (unset)   # NOT READ — the client sends the project per call
...
[env] ---- end agent server configuration ----

Process

Backend defaults

These name the default AI backend for calls whose profile does not choose one. A profile that names its own overrides them.

An unknown value in the environment is fatal, because a whole deployment quietly running a vendor nobody asked for is worse than refusing to boot. An unknown value in a request costs that one call and is reported — one caller’s typo should not take the fleet down.

Call defaults

Fallbacks for calls whose profile leaves the corresponding field unset.

Not read at all

These are printed at boot as NOT READ so a stale value cannot look load-bearing:

A call that arrives with no project or no key is refused with a message naming the field and both places it can be set. It is not quietly run against whatever ambient identity the host happens to carry — on a GCE instance that succeeds at authentication and then fails the Live handshake with insufficient authentication scopes, which describes neither the missing setting nor the account it borrowed.

5 - Interfaces

The image exposes two ports and nothing else. There is no web UI, no admin endpoint and no configuration API — the agent server is driven entirely by the agent client.

Port 8000 — health

One endpoint:

curl -s http://localhost:8000/health     # → ok

A liveness check, not a readiness one. It returns ok as soon as the process is up, including when no calls are running — which is the normal state. It does not tell you whether calls succeed; the log does that.

Safe to expose to a load balancer or monitoring system. It reveals nothing.

Port 50052 — control

This is where the agent client tells the server which room to join, which agent to run, and where to bill it. Everything about a call travels over this port.

It must be protected. Two reasons:

1. There is no encryption. The server holds no TLS certificate. The API key and the service-account credentials each call carries cross this port in clear text.
2. It controls live calls. A caller who reaches it can place agents into rooms and disconnect ones already talking.

The safe shapes are:

• Bind to 127.0.0.1 and run a TLS-terminating proxy in front — what the published examples do.
• Keep it on a private network that only the agent client can reach.

Publishing 0.0.0.0:50052 to the internet is not one of them.

Authentication

Every request must present ULAI_GRPC_API_KEY. The client is configured with the same value.

Leave the variable unset and the server accepts everything, including requests that disconnect live agents. It logs a warning saying so at startup, once:

[grpc] WARNING: ULAI_GRPC_API_KEY is unset — :50052 accepts unauthenticated
calls, including ones that disconnect live agents

One key, shared by every client. There is no per-client identity and no way to revoke one caller without rotating for all of them, so treat the key as an infrastructure secret rather than a per-team credential.

What it actually serves

For completeness, the port serves three gRPC services — AgentBridge (put an agent in a room, take it out), AgentSession (drive a live conversation) and AgentDispatch (receive calls as they start). Operating the server requires no knowledge of them; they matter to whoever builds the client.

Server reflection is enabled, so grpcurl can inspect the port for debugging — behind the same key:

grpcurl -H 'x-api-key: <key>' -plaintext localhost:50052 list

Outbound connections

The server also makes connections out, which your egress rules must allow:

Both are named per call by the client, not configured here.

6 - Operations

Running it, and finding out why it is not running.

6.1 - Deployment

The image

It is a heavy image and has to be: it owns the audio. RNNoise is compiled from vendored C, TEN VAD is a prebuilt .so, and libopus, libsoxr and libspeexdsp are linked for the transport and the DSP chain.

The runtime stage is Debian slim and carries only the binary plus those shared libraries. It owns no business data — no database, no agent store, no credentials.

Ports

Health

GET /health returns ok. It is a liveness check, not a readiness one — the process is healthy with zero calls running, which is the normal state.

Deployment order

The engine holds no project and no credentials, so the client that sends them must be deployed first. Deploying the server ahead of the client means every call is refused for want of a key.

The order:

1. Deploy the client with its placement configured.
2. Confirm [AIRoute] … creds=supplied (N bytes) on a call.
3. Deploy the server.

Reversing it produces a working-looking server and a fleet of refused calls.

Sizing

One process handles many concurrent calls; the limit is CPU, and the audio chain is what spends it. Each call runs denoise, foreground isolation and neural VAD every 20 ms.

GOMEMLIMIT is derived from the cgroup limit when unset — the boot log says what it picked:

[runtime] GOMEMLIMIT=13589MiB (/proc/meminfo limit=15987MiB, 85%)

Rolling restarts

Stop drains in-flight RPCs and then escalates to a hard stop after a timeout. One long-lived session stream must not hold a deploy open indefinitely — a session stream ends when its call does, which can be minutes away.

Calls in flight during a restart end. There is no session migration.

6.2 - Observability

Everything goes to stdout, so docker logs is the whole interface. Every line for a call carries its bridge id, which makes one call one filter:

docker logs agent_server --tail 100
docker logs agent_server 2>&1 | grep 48c7ad0bc30f     # one call
docker logs -f agent_server                            # follow

There are no metrics endpoints. Ship stdout to wherever you keep logs.

At boot

The whole configuration is printed, including variables that are set but not read — usually a rename:

[env] ---- agent server configuration ----
[env]   APP_GRPC_LISTEN_PORT   50052   # gRPC port; host is 127.0.0.1
[env]   ULAI_GRPC_API_KEY      set len=64 sha256:57a4cbf1
[env]   GEMINI_PROJECT_ID      (unset)   # NOT READ — the client sends it per call
[env]   NOT READ BY THIS PROCESS: APP_ENV APP_NAME SERVICE_HTTP_PORT
[env]   (set in the environment but nothing here looks at them — usually a rename)
[env] ---- end agent server configuration ----

Secrets are shown as a length and a hash, never a value.

Also at boot:

aibackend: realtime = gemini-live (default)
aibackend: classify = gemini-live (default)
[grpc] agentsession.v1 listening on [::]:50052 (AgentBridge served; AgentSession/AgentDispatch served)

Per call

[AIRoute] never prints a credential. It shows creds=supplied (2347 bytes) or creds=engine default — the latter meaning the profile carried none, which is now a refusal.

Turn accounting

played=…ms is not duration=…ms. It is how much agent audio the caller actually heard, and it differs from wall-clock by every second of silence, ringing and caller speech. It is the number that reflects what was delivered.

The same distinction appears per turn: a turn-ended event carries how much of that turn was heard before it was cut.

Audio

With AUDIO_NOTHING=true set on this host, the chain is cut down to the denoiser and says so loudly:

[AUDIO] AUDIO_NOTHING=true — pure RNNoise (preprocessor + foreground gate disabled)

That line in production is almost always a mistake — see configuration.

Prompts

PROMPT_LOG=on logs the system instruction at connect. Useful exactly once, when an agent is behaving oddly and you want to see what it was actually told. Noisy forever after, and it puts the prompt in your log store.

6.3 - Troubleshooting

Start here

docker ps --filter name=agent_server           # is it running?
curl -s http://localhost:8000/health           # → ok
docker logs agent_server --tail 100            # what happened

Every line for a call carries its bridge id, so one call is one filter:

docker logs agent_server 2>&1 | grep 48c7ad0bc30f

The call joins, then dies immediately

Look for backend connect failed: right after joined session=….

this call carried no credentials for project "…"

The profile carried a project but no key. The engine holds none of its own, so there is nothing to fall back onto.

Confirm it in this server’s log — creds=engine default on the [AIRoute] line means the call carried no key:

docker logs agent_server 2>&1 | grep AIRoute | tail -5

The fix is on the agent client, which is the thing that must send a key. Nothing you change on this host will help: it holds no credentials by design.

this call carried no AI project

Same shape, other field. The profile named neither a project nor a key.

insufficient authentication scopes

Seen on older builds that fell back to ambient credentials. The host’s own identity — a GCE instance service account — authenticated successfully and then failed the Live handshake, because it lacks the cloud-platform scope Vertex needs.

Current images refuse the call before this can happen, with a message naming the missing field. If you see this, you are on an older image — check the tag with docker inspect --format '{{.Config.Image}}' agent_server.

DetectAnsweringMachine needs a Classifier

A call asked for answering-machine detection on an image that could not supply a classifier for it. Current images turn detection off rather than failing the call. Upgrade the image, or have the client stop asking for AMD.

The agent joins but never speaks

Check for greeting primed. If it is absent, the backend never connected — see above. If it is present and there is still silence, the caller’s side is the place to look: the room roster line says whether anyone else is actually there.

An unknown backend kills the process at boot

aibackend: AI_BACKEND="gemini-liv" is not a known backend

Deliberate. A whole deployment quietly running a vendor nobody asked for is worse than refusing to start. The same value in a request costs only that call.

Calls are refused with unauthorized

The caller’s key does not match ULAI_GRPC_API_KEY. The boot log shows a hash of what the server expects:

[env]   ULAI_GRPC_API_KEY   set len=64 sha256:57a4cbf1

If it says (unset), the server is accepting everything, and the failure is elsewhere.

A tool never runs

Tools are configured on the agent and answered by the agent client — there is nothing to enable on this host. A tool that never runs is a client-side problem.

A call that pauses, goes quiet and then apologises is usually a tool the client answered too slowly. See tools.

Captions stop after the first utterance

Every caller utterance landing in one caption is a segment that never closed. Fixed in current images; on an older one the caller’s transcript accumulates into a single bubble while the agent’s lines appear separately. Upgrade.

The caller sounds gated or clipped

Have the client send audio_nothing on one call, or set AUDIO_NOTHING=true on this host to test every call. It strips the chain to the denoiser alone, so if the problem disappears it is the pre-clean stage or the near-field foreground gate.

It is a diagnostic, not a setting. The full chain is the tuned one, and the stage it removes is what keeps a noisy room out of the model.