Documentation
Ulai is built from independent modules. Each one is documented on its own, with
its own configuration, API and operational notes.
Modules
Agent Server
ulai-agent-server — the voice engine, shipped as a
container image. Joins an Ulai SFU room, connects an AI backend, and runs the
conversation: the audio chain, turn detection, barge-in and the end-call
contract. It holds no database and no credentials — everything about how a call
runs arrives with the call.
Overview ·
Deploy it ·
Configuration ·
Interfaces
More modules are documented here as they land.
1 - SIP Gateway
SIP trunks in, Ulai SFU voice rooms out.
ulai-sip-module is a single Go binary (sip-sfu-gateway) that puts a phone
call into an Ulai SFU room. A carrier’s INVITE is routed, admitted and
answered; an API call dials out through a trunk. Either way the caller ends up
as an ordinary WebRTC participant talking to whoever else is in the room.
Source: github.com/ulaidotin/ulai-sip-module
Run it
docker run -d --name ulai-sip \
-e ULAI_CONTROL_PLANE_URL=https://stgcp.ulai.co.in \
-e SIP_ROUTING_REDIS_URL='redis://USER:PASSWORD@redis.ulai.co.in:6379' \
-e SIP_PUBLIC_IP=148.113.58.51 \
-e SIP_LISTEN_ADDR=0.0.0.0:5060 \
-e SIP_TRANSPORT=udp \
-e SIP_RTP_HOST=0.0.0.0 \
-e SIP_RTP_PORT_LOW=10000 \
-e SIP_RTP_PORT_HIGH=10500 \
-e SIP_HTTP_PORT=8082 \
--network host \
asia-south1-docker.pkg.dev/arctic-operand-415316/ulai/sip:v6
--network host is not optional — see
networking.
Dial out
curl -X POST http://148.113.58.51:8082/sip/originate \
-H 'Authorization: Bearer ulai_live_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' \
-H 'Content-Type: application/json' \
-d '{
"to_number": "+919363399639",
"trunk_id": "74958094-8c76-419f-aa42-d4eac6768ec2",
"project_id": "fa3f23f8-a75d-44a4-84f0-6a1c828f579f",
"from_number": "+13187184515",
"name": "sip-module",
"max_participants": 8,
"offer_srtp": true,
"require_srtp": true,
"dial_verbatim": false,
"session_id": "527f78a3262dbe8b0d0a678a9e98ac29"
}'
In this section
- Overview — what it does, what it does not do, and
where it sits in the platform.
- Getting started — prerequisites, build,
configuration, and your first inbound and outbound call.
- Concepts — call flows, the routing plane, and the
media pipeline.
- Configuration — every environment variable and
built-in timeout.
- HTTP API —
/health, /calls and
/sip/originate. - Operations — deployment, networking,
observability and troubleshooting.
1.1 - Overview
What the gateway does, and where it sits in the platform.
What it is
The Ulai SIP Gateway is a single Go binary (sip-sfu-gateway) that joins two
worlds:
- SIP/RTP — a carrier trunk, speaking SIP over UDP, TCP or TLS, carrying
G.711 audio over RTP or SRTP.
- Ulai SFU rooms — WebRTC sessions on the Ulai control plane, where browsers
and AI agents meet.
A phone call that reaches the gateway becomes an ordinary participant in a
room. Everything the gateway does is in service of that one sentence: decide
whether the call is allowed, get a room for it, answer it, and move audio in
both directions until somebody hangs up.
It handles both directions of origination:
| Direction | Trigger | The gateway is |
|---|
| Inbound | A carrier sends an INVITE to the SIP listener | the answering party (UAS) |
| Outbound | POST /sip/originate with a number and a trunk id | the calling party (UAC) |
What it is not
- Not an agent. The gateway never talks to a model, never decides what to
say, and has no notion of a conversation. It publishes
CALL_ANSWERED and
lets the platform’s dispatcher summon whatever should join the room. - Not a media processor. No noise suppression, no VAD, no recording, no
playback accounting. It transcodes and it forwards.
- Not a rate limiter. It dials on every
POST /sip/originate. If a trunk
answers 429, the caller upstream is the one that has to slow down. - Not a routing database. Numbers, trunks, ACLs and dispatcher rules live in
the platform’s routing store; the gateway only reads them.
Where it sits
carrier / ITSP gateway Ulai platform
┌───────────────┐ ┌──────────────────┐ ┌──────────────────┐
│ SIP trunk │──────▶ │ SIP listener │ │ control plane │
│ (udp/tcp/tls) │ INVITE │ :5060 │──────▶ │ create / join / │
│ │ │ │ HTTPS │ terminate │
│ │◀───── │ RTP :10000-10500 │ └──────────────────┘
└───────────────┘ RTP │ │ ┌──────────────────┐
│ HTTP API :8082 │──────▶ │ routing store │
│ /sip/originate │ Redis │ (sip-resolver) │
│ /calls /health │ └──────────────────┘
│ │ ┌──────────────────┐
│ WebRTC / Opus │──────▶ │ SFU room │
└──────────────────┘ SRTP │ agent, browsers │
└──────────────────┘
Three external dependencies, all required:
- The control plane (
ULAI_CONTROL_PLANE_URL) creates rooms, issues join
tickets, and feeds the roster and lifecycle events for a session. - The routing store (
SIP_ROUTING_REDIS_URL) is read through
ulai-sip-resolver. It is
the authority on which project owns a number, which source IPs a trunk
accepts, which dispatcher rule matches, and how to reach an outbound trunk.
It is also where telecom events are published. - The carrier, reachable at
SIP_PUBLIC_IP.
Why a separate service
The SIP stack and the room gateway share no dependencies: the SIP side links
no WebRTC, and the room side links no SIP. That keeps the binary — and the
image — small, and it is why the only native dependency in the whole thing is
libopus, because there is no production-grade pure-Go Opus encoder. Every
other piece of the audio path — resampling, μ-law/A-law, RTP, SDP, SRTP — is
pure Go.
Where to go next
1.2 - Getting started
Run the image and put a call through it.
The gateway ships as a container image. There is nothing to compile and nothing
to install on the host beyond Docker.
What you need
| Requirement | Why |
|---|
| A host with a routable public IP | It is advertised to the carrier in SDP and Contact, and media is streamed to it |
| Docker, with host networking available | The RTP range cannot be published through a proxy — see networking |
| A control-plane URL | Rooms are created and joined there |
| A routing store URL (Redis) | Every call is resolved against it |
| A SIP trunk configured in the routing store | Inbound numbers, outbound termination, or both |
| A control-plane API key | Required for POST /sip/originate |
Ports that must be reachable from the carrier: 5060 (UDP, TCP or TLS) for
signalling and 10000–10500 (UDP) for media. Opening the first without the
second gives you calls that connect and then have no audio.
Pull the image
docker pull asia-south1-docker.pkg.dev/arctic-operand-415316/ulai/sip:v6
If the registry is private, authenticate first:
gcloud auth configure-docker asia-south1-docker.pkg.dev
Run it
docker run -d --name ulai-sip \
-e ULAI_CONTROL_PLANE_URL=https://stgcp.ulai.co.in \
-e SIP_ROUTING_REDIS_URL='redis://USER:PASSWORD@redis.ulai.co.in:6379' \
-e SIP_PUBLIC_IP=148.113.58.51 \
-e SIP_LISTEN_ADDR=0.0.0.0:5060 \
-e SIP_TRANSPORT=udp \
-e SIP_RTP_HOST=0.0.0.0 \
-e SIP_RTP_PORT_LOW=10000 \
-e SIP_RTP_PORT_HIGH=10500 \
-e SIP_HTTP_PORT=8082 \
--network host \
asia-south1-docker.pkg.dev/arctic-operand-415316/ulai/sip:v6
Three things in that command are load-bearing:
--network host is required. Publishing the RTP range with -p spawns
hundreds of userland proxies and makes every INVITE arrive from the bridge
gateway’s address, which the trunk’s IP ACL then rejects with 403.SIP_PUBLIC_IP must be the host’s own routable address. It is what every
SDP tells the carrier to stream media to, which is why 0.0.0.0 is rejected
at startup.SIP_RTP_HOST=0.0.0.0 is what the RTP sockets bind to. On most cloud
VMs the public IP is attached at a NAT or load-balancer layer and is not on
any interface inside the VM, so binding to it fails outright.
Keep secrets out of the host’s process list with an env file instead:
docker run -d --name ulai-sip --network host \
--env-file /etc/ulai/sip.env \
asia-south1-docker.pkg.dev/arctic-operand-415316/ulai/sip:v6
# /etc/ulai/sip.env
ULAI_CONTROL_PLANE_URL=https://stgcp.ulai.co.in
SIP_ROUTING_REDIS_URL=redis://USER:PASSWORD@redis.ulai.co.in:6379
SIP_PUBLIC_IP=148.113.58.51
SIP_LISTEN_ADDR=0.0.0.0:5060
SIP_TRANSPORT=udp
SIP_RTP_HOST=0.0.0.0
SIP_RTP_PORT_LOW=10000
SIP_RTP_PORT_HIGH=10500
SIP_HTTP_PORT=8082
Only three variables are genuinely required — ULAI_CONTROL_PLANE_URL,
SIP_ROUTING_REDIS_URL and SIP_PUBLIC_IP. Everything else has a default.
Every knob is in the configuration reference.
Check it
A healthy start looks like this:
[runtime] GOMAXPROCS=4 (cgroup CPU quota=4.00, host cores=8)
[runtime] GOMEMLIMIT=3481MiB (cgroup limit=4096MiB, 85%)
[runtime] GOGC=200
routing store reachable
sip-sfu-gateway: SIP listening on 0.0.0.0:5060/udp public=148.113.58.51 rtp=10000-10500 rtp_timeout=30s
sip-sfu-gateway: HTTP listening on :8082 (control plane https://stgcp.ulai.co.in, ice_servers=1)
If the routing store cannot be reached the gateway still starts, and says so:
WARNING: routing store unreachable (...) — every call will be rejected until it recovers
That is deliberate — a listener that answers with an honest 500 is worth more
than one that is not there at all.
Then the two endpoints:
curl -s localhost:8082/health
# {"status":"ok","service":"sip-sfu-gateway"}
curl -s localhost:8082/calls
# {"live":[]}
/health says nothing about the SIP leg. Check that separately — the image
ships sipsak for exactly this:
docker exec ulai-sip sipsak -s sip:healthcheck@127.0.0.1:5060
Your first inbound call
- Point a number at the gateway in the routing store: the SIP domain the
carrier addresses must map to a project, the number must be assigned, the
trunk’s IP ACL must include the carrier’s source address, and a dispatcher
rule must match.
- Dial the number.
- Watch the log. Each line of a call is prefixed with the number, so a busy
gateway can be read one call at a time:
[+919000000000] [sip-gw] INVITE: to=+919000000000 from=+919111111111 host=sbc.example.com source=13.14.15.16 callID=3e114ad4…
[+919000000000] [sip-gw] routed: project=proj_123 trunk=Main SBC rule=support (direct) agent=support-bot
[+919000000000] [sip-gw] room sess_abc (via resolver, created=true)
[+919000000000] [sip-gw] call accepted — connecting to room sess_abc
[+919000000000] [sfugw] webrtc state: connected
A rejection is equally legible, and the SIP status says which check failed —
see troubleshooting.
Your first outbound call
POST /sip/originate needs a control-plane API key and a trunk id. The room is
created for you unless you pass session_id:
curl -X POST http://148.113.58.51:8082/sip/originate \
-H "Authorization: Bearer $ULAI_API_KEY" \
-H 'Content-Type: application/json' \
-d '{
"to_number": "+919363399639",
"trunk_id": "74958094-8c76-419f-aa42-d4eac6768ec2",
"project_id": "fa3f23f8-a75d-44a4-84f0-6a1c828f579f",
"from_number": "+13187184515",
"name": "sip-module",
"max_participants": 8,
"offer_srtp": true,
"require_srtp": true,
"dial_verbatim": false
}'
Only to_number and trunk_id are required, plus project_id unless
ULAI_PROJECT_ID is set. Everything else has a default — including the SRTP
policy, which is inferred from the trunk’s transport. Add
"session_id": "527f78a3262dbe8b0d0a678a9e98ac29" to dial into a room that
already exists instead of creating one.
The response comes back immediately:
{
"status": "originating",
"to_number": "+919363399639",
"session_id": "527f78a3262dbe8b0d0a678a9e98ac29",
"created": true,
"name": "sip-module",
"trunk": "Main SBC"
}
The 202 comes back as soon as the trunk is resolved and the room exists — the
phone is still ringing. Point an agent or a browser at session_id while it
does. The room is only joined once the callee actually answers, so nothing
greets a ringtone.
Stopping it
SIGTERM starts a drain, not a kill: new calls are refused, in-flight calls get
up to 60 seconds to finish, and anything left is then cancelled with 5 seconds
of grace. Give Docker a matching timeout so it does not SIGKILL through it:
docker stop --time 70 ulai-sip
Where to go next
- Concepts — what happens between the INVITE and the audio.
- HTTP API — the full request and response shapes.
- Operations — networking, observability, troubleshooting.
1.3 - Concepts
How the gateway is put together and how a call moves through it.
Three things are worth understanding before changing anything:
- Call flows — the ordered steps of an inbound
and an outbound call, and why the order differs.
- Routing plane — how a call is admitted, how a
trunk is resolved, and what gets published.
- Media pipeline — codecs, SRTP, jitter, DTMF, and
the ways a call ends when nobody sends a
BYE.
1.3.1 - Call flows
The ordered steps of an inbound and an outbound call — and why the orders differ.
Both legs end in the same place: a bridgedCall handed to sfugw.Run, which
moves audio until one side stops. Everything before that differs.
Inbound
The gateway is the answering party. telephony.SIPRuntime invokes
gateway.OnInvite for every out-of-dialog INVITE.
carrier gateway resolver control plane
│ INVITE ────────▶ │ │ │
│ │ 1. Resolve ───────────▶│ │
│ │ (host, number, IP) │ │
│ ◀── 4xx/5xx ──── │ ◀── reject ────────────│ │
│ │ │ │
│ │ 2. CreateSession ──────┼────────────────▶│
│ ◀── 200 OK ───── │ 3. Accept │
│ ──── ACK ──────▶ │ │
│ │ 4. PublishEvent CALL_ANSWERED ─▶ resolver│
│ │ 5. JoinSession ────────┼────────────────▶│
│ ◀═══ RTP ══════▶ │ ◀════════ bridge ══════╪═════ WebRTC ═══▶│
- Route.
(host, dialled number, source IP) → project, trunk ACL,
dispatcher rule. A rejection here is a specific SIP status the carrier can
act on, not a blanket 503. See routing. - Room. An
X-Session-Id header on the INVITE wins if present; otherwise
a session is created, carrying the dispatcher rule in its metadata so the
platform can see what the room is for. The header only chooses the room — it
cannot overrule admission. - Answer.
200 OK + ACK, which binds RTP. - Tell.
CALL_ANSWERED is published before the WebRTC handshake, not
after. That event is what summons the agent, so the dispatcher gets to work
while the gateway is still joining. - Join and bridge. A seat is taken in the room, then audio flows.
The room is joined only after the call is answered. On this path “accepted”
is the pickup, so no seat is taken until there is a live call to put in it —
a join ticket is only valid for about 30 seconds, and a participant appearing in
the room is the agent’s cue to start talking.
Rejections
| Condition | SIP status |
|---|
Empty To user | 400 Bad Request |
| Gateway is shutting down | 503 Shutting down |
| Unknown domain, unassigned number, missing config | 404 Not Found |
| Source IP not in the trunk’s ACL | 403 Forbidden |
| No dispatcher rule matched | 480 Temporarily Unavailable |
| Routing lookup failed | 500 Server Internal Error |
| Session could not be created | 503 Session unavailable |
The project-id lookup is deliberately best-effort: a call that routed cleanly is
not dropped because that second lookup missed. The dispatcher loses a field, the
caller keeps their call.
Outbound
The gateway is the calling party. POST /sip/originate splits into a
synchronous half the caller can act on and an asynchronous half it cannot.
caller gateway resolver control plane
│ POST ──────────▶ │ │ │
│ │ auth: Bearer <ulai key> │ │
│ │ 1. GetOutboundTrunk ───▶│ │
│ ◀── 404/502 ──── │ ◀── not found ──────────│ │
│ │ 2. CreateSession ───────┼────────────────▶│
│ ◀── 202 ──────── │ (unless session_id given) │
│ {session_id} │ │
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌ │ ╌╌ background ╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌ │
│ │ 3. INVITE ──▶ trunk, block through ring │
│ │ 4. PublishEvent CALL_ANSWERED ─▶ resolver │
│ │ 5. JoinSession ─────────┼────────────────▶│
│ │ ◀════════ bridge ═══════╪═════ WebRTC ═══▶│
- Trunk first. An unknown
trunk_id must not leave an orphan room behind,
so the trunk is resolved before anything is created. - Room. Created unless the request carried a
session_id. Its metadata
carries the number, the trunk and the agent_id, so an orchestrator watching
the control plane’s discovery feed can be ready before the phone is answered. - Dial.
Originate blocks through ring, up to OriginateTimeout (60 s).
The dial context stays alive for the whole call — sipgo builds the client
transaction on it, so cancelling it at answer would pull the dialog out from
under the call just connected. - Tell, then 5. join and bridge, exactly as inbound.
Ringing takes up to 30 seconds and has nothing useful to say until it is over,
which is why it is not on the request path. Trunk lookup and room creation are
fast, can fail in ways the caller can act on, and produce a result the caller
needs — so they are.
If the callee never answers
Busy, rejected, no answer, unreachable trunk — the room created for the call is
terminated, because anything already waiting in it deserves to be told rather
than left listening to silence. A 429 from the trunk is logged specially: it
is the one failure that is the platform’s own fault, and it is fixed by dialling
slower, not by retrying.
What both legs do at the end
sfugw.Run returns when the SIP leg drops, the room ends, or the context is
cancelled. Then:
CALL_HANGUP is published, with duration_seconds, pairing the
CALL_ANSWERED already sent. It is published on a context detached from the
call’s own, because the call’s context is usually being torn down at exactly
that moment.- The call is removed from
GET /calls. - The session is terminated, unless the room was borrowed and
SIP_TERMINATE_SESSION_ON_HANGUP=false.
1.3.2 - Routing plane
How a call is admitted, how a trunk is resolved, and what gets published.
Everything the gateway knows about numbers, trunks and agents comes from
ulai-sip-resolver over the
platform’s Redis. The gateway holds no routing table of its own.
inbound : (host, dialled number, source IP) → project, trunk ACL, dispatcher rule
outbound : (project, trunk id) → address, transport, digest auth
Neither leg carries trunk details on the wire. Callers used to send host, port,
transport and credentials on every origination, which put carrier passwords in
every dialler’s logs and meant a trunk migration had to be rolled out to each of
them.
Inbound admission
Resolve(host, dialledNumber, sourceIP) is the whole admission decision. It
returns the trunk the call came in on and the dispatcher rule that won, or an
error that maps to a specific SIP status:
| Resolver error | Status | Means |
|---|
ErrHostNotFound | 404 | No project owns that SIP domain |
ErrNumberNotAssigned | 404 | The domain is known, the number is not |
ErrConfigNotFound | 404 | The project has no SIP configuration |
ErrIPNotAllowed | 403 | The INVITE came from an address the trunk’s ACL does not list |
ErrNoMatchingRoute | 480 | Admitted, but no dispatcher rule matched the number |
| anything else | 500 | The lookup itself failed |
Each one is a different operational problem and deserves to be distinguishable
from the carrier’s side.
The project id is a second, separate lookup (ulai:sip_host:<host>), because
the resolver’s Result does not carry it. That lookup is best-effort: if it
fails the call still proceeds, logged as continuing without a project id, and
the published events lose a field.
A room created for an inbound call carries the routing decision in its metadata,
which rides the control plane’s discovery feed. An orchestrator that has never
heard of the call can read it and know which agent the room is waiting for:
| Key | Source |
|---|
project_id | The host lookup |
trunk_id, trunk_name | The resolved trunk |
rule_id, rule_name, rule_type | The dispatcher rule |
agent_name | The rule, when it names one |
dispatch_metadata | The rule’s free-form metadata, JSON-encoded |
name, phone_number, dialled, sip_call_id | The call |
direction | inbound |
source | sip-sfu-gateway |
Empty fields are left out rather than written blank. Several of them
routinely are — rules are stored keyed by id with no id inside the value — and a
dispatcher can act on an absent rule_id where it cannot tell a blank one from
a real empty answer.
An outbound room’s metadata is the same idea with the outbound fields:
phone_number, caller_id, name, agent_id, project_id, trunk_id,
trunk_name, direction: outbound, source.
Outbound trunks
GetOutboundTrunk(projectID, trunkID) returns a stored trunk, which the gateway
reduces to dial parameters:
- Address — accepted as
host, host:port, or either with a sip:/sips:
scheme, userinfo or URI parameters attached. Port 0 means unspecified, and
the dialler fills in 5061 for TLS or 5060 otherwise. - Transport —
udp, tcp, tls, or empty (treated as UDP). Anything else
is an error rather than a silent fallback: dialling a TLS-only trunk over UDP
fails as a timeout minutes later, which is a miserable way to learn about a
typo. - Credentials — digest username and password, empty when the trunk
authenticates by IP ACL.
The stored trunk carries a transport but no media-encryption field, so SRTP
policy is derived:
| Trunk transport | Offer SRTP | Require SRTP |
|---|
tls | yes | no |
udp, tcp | no | no |
This is not cosmetic. A carrier with secure trunking enabled — Twilio’s is, on a
TLS trunk — answers an RTP/AVP offer with 488 Secure media required and the
call never rings.
Offering without requiring is the safe half: a carrier that wants SRTP finds
crypto in the offer; one that does not echoes no a=crypto and the call falls
back to plain RTP. Override per request with offer_srtp and require_srtp —
including offer_srtp: false to force cleartext media on a TLS trunk.
require_srtp implies offer_srtp; asking for the contradictory pair
(offer_srtp: false, require_srtp: true) is a 400.
Events
Two event types are published to the platform’s telecom stream, through the same
resolver that authorised the call:
| Type | When | Notable metadata |
|---|
CALL_ANSWERED | Immediately after the call is answered, before the room is joined | routing fields, session_id, sip_call_id, direction, to_number, from_number |
CALL_HANGUP | When the bridge ends | the same, plus duration_seconds |
Publishing is bounded at 2 seconds and best-effort: an event is never allowed to
hold up a call, and a miss is logged and nothing more. The hangup publish is
detached from the call’s context on purpose — that context is being cancelled at
precisely the moment the event matters most.
Startup probe
go-redis connects lazily, so without a probe the first sign of a dead routing
store would be a carrier receiving a 500. At startup the gateway looks up a
host no project can own and reports what happened:
routing store reachable
WARNING: routing store unreachable (...) — every call will be rejected until it recovers
It does not fail startup. A malformed URL already did that; a store that is
down now may be up a second from now.
1.3.3 - Media pipeline
Codecs, SRTP, jitter, DTMF, and the ways a call ends without a BYE.
The audio path
The carrier speaks 8 kHz G.711. The SFU speaks 48 kHz Opus. Everything between
the two happens inside the gateway:
uplink caller μ-law 8k ─decode─▶ PCM 8k ─resample─▶ PCM 48k ─encode─▶ Opus ─▶ room
downlink room Opus 48k ─decode─▶ PCM 48k ─resample─▶ PCM 8k ─mix─▶ μ-law ─▶ caller
Both directions run on a 20 ms cadence: 160 μ-law bytes on the wire, 960
samples per Opus frame. Opus is encoded at 24 kbit/s with complexity 5 —
wideband speech without maxing out a CPU that may be carrying many concurrent
calls.
The downlink mixes: a room can hold several publishers, and the caller gets
all of them summed into one stream. Resampling is go-audio-resampler (pure
Go, SIMD-enabled); μ-law and A-law conversion is zaf/g711. The only native
code in the binary is libopus.
Codec negotiation
The SDP profile is deliberately narrow — the audio pipeline is μ-law 8 kHz, so
accepting anything else would be a lie:
- PCMU (μ-law, PT 0) or PCMA (A-law, PT 8). When the carrier negotiates
A-law, the RTP loops transcode A-law↔μ-law at the wire boundary so everything
above stays μ-law.
- telephone-event (RFC 4733 DTMF) on whatever payload type the carrier
assigns.
- No Opus on the SIP side, no video, no multiple
m= lines.
SRTP
SDES only, with the two common profiles: AES_CM_128_HMAC_SHA1_80 (Twilio’s
default) and _32. Whether it is offered is decided by the trunk’s transport —
see routing.
Inbound RTP: reorder, dedupe, conceal
A conventional jitter buffer imposes a fixed delay on every packet. For a voice
agent that delay is charged to time-to-first-token on every turn, including
the overwhelming majority where the network was perfectly ordered. So the
inbound buffer is not a fixed-delay design:
| Packet arrives | What happens |
|---|
| In sequence | Emitted immediately, zero added latency |
| Out of order | Held only until the gap resolves, bounded by a 40 ms holdout |
| Duplicated | Dropped |
| Never | Concealed after the holdout, stream continues |
Concealment repeats the previous frame, attenuated, decaying to silence over a
few frames. Repeating preserves the spectral envelope so an ASR hears a brief
smear rather than the click-and-jump digital silence produces; decaying stops a
lost burst becoming an audible buzz.
The cost is paid only by calls that actually have a disordered network.
Outbound RTP: a small playout cushion
The provider’s writer and the RTP writer are two independent 20 ms tickers.
Without a cushion, ordinary scheduler jitter forces silence into the middle of
speech. The playout buffer builds 60 ms (3 frames) before starting and caps
added latency at 160 ms (8 frames), dropping the oldest beyond that.
Symmetric RTP and the source gate
Carriers behind NAT routinely send RTP from a port they never advertised in
SDP, so the first well-formed packet latches the peer address and the writer
re-targets to it. Everything after that is checked against the latch.
That check matters because the RTP port range is a few hundred even ports cycled
round-robin: a call that ends while its carrier is still streaming leaves
packets in flight that land on whichever call binds that port next. Before the
gate existed, they were decoded and mixed into a live conversation as a second
voice.
A genuine media re-anchor — a B2BUA leg swap, an SBC failover — is admitted
only after the new source proves persistence (5 packets over at least 200 ms).
A re-INVITE can pre-authorise an address, but the previous peer stays valid
until the new one actually speaks: a re-INVITE is an intention, and a peer that
never follows through must not be able to mute a working call.
Re-INVITE, hold, and session timers
sipgo’s OnInvite fires for every INVITE, including in-dialog ones. Handing
those to the application handler treats a mid-call re-INVITE — a session-timer
refresh, a hold, a media re-anchor, an SBC failover — as a brand new call:
a second agent, a second billing row, a 180 Ringing inside an established
dialog, a 200 OK with a new To tag (a protocol violation), and an answer
advertising a new RTP port, so the carrier moves media to a socket nobody reads.
Dead air for the rest of the call.
In-dialog INVITEs are therefore handled separately, and answered as a
re-statement rather than a negotiation: same To tag, same RTP port, same
codec, session id unchanged with its version bumped, and the direction attribute
mirrored so hold is acknowledged rather than contradicted. The only thing that
may legitimately change is where the peer wants media — and that is followed.
A re-INVITE that tries to switch G.711 flavour mid-call is refused rather than
silently answered with a lie: the provider’s transcode setting is fixed when it
is constructed.
Session timers (timer) are supported; an INVITE that Requires an extension
the gateway does not support is rejected rather than answered.
DTMF
RFC 4733 telephone-event packets are decoded and surfaced as events. The
gateway logs them (DTMF: 5) and does nothing else with them — it has no IVR of
its own. An agent in the room that wants digits should consume the session
feed, not expect the gateway to act.
Ending a call without a BYE
A hangup’s BYE can be lost. A TLS trunk calling back a UDP-only listener never
reaches the gateway at all, and without a backstop such a call — and its room —
stays up until the process exits.
SIP_RTP_TIMEOUT_SECONDS (default 30) ends a call whose inbound audio has
stopped for that long, as if the far end had hung up. It is paused while the
call is on hold, so a legitimately silent leg is not cut off. Set 0 to
disable.
The other end-of-call signals:
| Signal | Source |
|---|
BYE | The far end, normally |
session_terminated | The control plane, over the session events feed — the authoritative “this call is really over” |
Transport Done() | The WebRTC leg dropping, which may be a transient blip the SDK reconnects through |
| Drain cancellation | Shutdown, after the 60 s drain window |
1.4 - Configuration
Every environment variable, its default, and what breaks when it is wrong.
Configuration is read from the environment once, at startup, into a single
validated struct. Everything downstream reads that struct, never the
environment — which is also why a test can construct one directly.
Validation reports every problem at once, so a misconfigured deployment does
not have to be fixed one restart at a time.
Required
| Variable | Description |
|---|
ULAI_CONTROL_PLANE_URL | Base URL of the Ulai control plane, e.g. https://stgcp.ulai.co.in. Rooms are created, joined and terminated here. |
SIP_ROUTING_REDIS_URL | The routing store every call is resolved against. REDIS_URL is accepted as a fallback. |
SIP_PUBLIC_IP | The address advertised in SDP and Contact. Must be routable by the carrier. 0.0.0.0 is rejected — it is not a listen address, it is what the carrier is told to stream media to. |
SIP signalling
| Variable | Default | Description |
|---|
SIP_LISTEN_ADDR | 0.0.0.0:5060 | What sipgo binds to. host:port; the host may be 0.0.0.0. |
SIP_TRANSPORT | udp | udp, tcp or tls. |
SIP_TLS_CERT_PATH | — | Required when SIP_TRANSPORT=tls. |
SIP_TLS_KEY_PATH | — | Required when SIP_TRANSPORT=tls. |
SIP_SESSION_HEADER | X-Session-Id | The INVITE header that may point a call at an existing room. Must not be blank. |
| Variable | Default | Description |
|---|
SIP_RTP_HOST | 0.0.0.0 | What RTP sockets bind to, as distinct from SIP_PUBLIC_IP, which is what is advertised. On most cloud VMs the public IP is attached at a NAT or load-balancer layer and is not on any interface inside the VM, so binding to it fails with cannot assign requested address. The default is right nearly everywhere. |
SIP_RTP_PORT_LOW | 10000 | Bottom of the RTP allocation range. |
SIP_RTP_PORT_HIGH | 10500 | Top of the range. Must be above the low end, and both within 1–65535. Even ports only, by RTCP convention. |
SIP_RTP_TIMEOUT_SECONDS | 30 | End a call whose inbound audio has stopped for this long, as if the far end had hung up — the backstop for a BYE that never arrives. Paused while the call is on hold. 0 disables it. |
HTTP API
| Variable | Default | Description |
|---|
SIP_HTTP_PORT | 8082 | Port for /health, /calls and /sip/originate. Chosen clear of the worker (8080) and the Tata gateway (8081). |
Rooms
| Variable | Default | Description |
|---|
ULAI_PROJECT_ID | — | Default project for POST /sip/originate when the body omits project_id. The single-tenant deployment’s answer to sending it every time. |
ULAI_MAX_PARTICIPANTS | 8 | Size of a room the gateway creates. Overridden per request by max_participants. Must be positive. |
SIP_TERMINATE_SESSION_ON_HANGUP | true | End the whole session — every participant, every transport — when the SIP leg drops, rather than merely leaving the room. Set false only when a room reached via X-Session-Id or session_id is genuinely shared. A room the gateway created is always terminated regardless. |
ICE
The gateway is STUN-only by default (stun:stun.l.google.com:19302), which is
fine on a public host and will fail behind symmetric NAT.
| Variable | Default | Description |
|---|
TURN_URLS | — | Comma-separated TURN URLs. Appended to the STUN default. |
TURN_USERNAME | — | TURN credential. |
TURN_PASSWORD | — | TURN credential. |
Go runtime
At startup the process aligns the Go runtime with its cgroup limits — two
defaults that are correct on a bare host become latency or OOM bugs inside a
container. An explicit environment variable always wins.
| Variable | Behaviour when unset |
|---|
GOMAXPROCS | Pinned to ceil(cgroup CPU quota) when that is below the host core count. Left alone otherwise. |
GOMEMLIMIT | Set to 85% of the cgroup memory limit, or of total system RAM when there is none. The headroom covers goroutine stacks and the off-heap libopus and resampler allocations GOMEMLIMIT cannot see. |
GOGC | Raised to 200, trading memory for fewer GC cycles and less latency jitter, bounded by the limit above. |
Each decision is logged at startup:
[runtime] GOMAXPROCS=4 (cgroup CPU quota=4.00, host cores=8)
[runtime] GOMEMLIMIT=3481MiB (cgroup limit=4096MiB, 85%)
[runtime] GOGC=200
Built-in timeouts
Not configurable — they are constants, listed here because they explain the
behaviour you will observe.
| Constant | Value | Bounds |
|---|
ResolveTimeout | 5 s | One routing-store lookup |
CreateTimeout | 15 s | Creating a room |
JoinTimeout | 15 s | Joining a room |
DialTimeout | 15 s | The WebRTC signalling dial |
OriginateTimeout | 60 s | An outbound INVITE, through ring |
TerminateTimeout | 5 s | Terminating a room, or hanging up a leg |
DrainTimeout | 60 s | The shutdown drain |
CancelGrace | 5 s | Extra time after the drain deadline cancels calls |
MaxRequestBytes | 8 KiB | An originate body — it is a control API, not an upload |
| publish timeout | 2 s | One event publish, best-effort |
| control-plane HTTP timeout | 10 s | Any single control-plane request |
Example
# Required
ULAI_CONTROL_PLANE_URL=https://stgcp.ulai.co.in
SIP_ROUTING_REDIS_URL=redis://user:password@redis.example.com:6379/0
SIP_PUBLIC_IP=203.0.113.10
# SIP + RTP
SIP_LISTEN_ADDR=0.0.0.0:5060
SIP_TRANSPORT=udp
SIP_RTP_PORT_LOW=10000
SIP_RTP_PORT_HIGH=10500
SIP_RTP_TIMEOUT_SECONDS=30
# HTTP
SIP_HTTP_PORT=8082
# Rooms
SIP_TERMINATE_SESSION_ON_HANGUP=true
1.5 - HTTP API
/health, /calls and /sip/originate.
Three endpoints, served on SIP_HTTP_PORT (default 8082). Only
/sip/originate is authenticated.
| Method | Path | Auth |
|---|
GET | /health | none |
GET | /calls | none |
POST | /sip/originate | Authorization: Bearer <ulai api key> |
These endpoints are not public
/health and /calls are unauthenticated, and /calls lists live phone
numbers. Keep the HTTP port on a private network or behind a firewall; only
/sip/originate checks a credential.
GET /health
curl -s http://148.113.58.51:8082/health
{ "status": "ok", "service": "sip-sfu-gateway" }
Always 200 while the process is serving. It says nothing about the SIP leg or
the routing store — see observability for
a check that covers both.
GET /calls
Every call currently bridged, oldest first.
curl -s http://148.113.58.51:8082/calls
{
"live": [
{
"to_number": "+919363399639",
"from_number": "+13187184515",
"name": "sip-module",
"sip_call_id": "3e114ad4-a9ad-4a0b-97b9-5cf11190bdd8",
"session_id": "527f78a3262dbe8b0d0a678a9e98ac29",
"project_id": "fa3f23f8-a75d-44a4-84f0-6a1c828f579f",
"direction": "outbound",
"source": "originate",
"started_at": "2026-09-21T10:14:52.118Z"
}
]
}
| Field | Meaning |
|---|
to_number, from_number | Normalised E.164 |
name | Display name in the room roster |
sip_call_id | The SIP Call-ID — the join key between logs, events and carrier CDRs |
session_id | The Ulai room |
project_id | Owning project; may be empty inbound if the host lookup missed |
direction | inbound or outbound |
source | originate, resolver, or the session header’s name when a room was supplied on the INVITE |
started_at | When the bridge started, RFC 3339 |
The list is sorted rather than map-ordered, so polling it does not reshuffle
rows on every request.
POST /sip/originate
Places one outbound call. Authenticates with a control-plane API key, and
the room opened for the call belongs to the project that key identifies — which
is how one gateway serves several projects without per-project configuration.
curl -X POST http://148.113.58.51:8082/sip/originate \
-H 'Authorization: Bearer ulai_live_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' \
-H 'Content-Type: application/json' \
-d '{
"to_number": "+919363399639",
"trunk_id": "74958094-8c76-419f-aa42-d4eac6768ec2",
"project_id": "fa3f23f8-a75d-44a4-84f0-6a1c828f579f",
"from_number": "+13187184515",
"name": "sip-module",
"max_participants": 8,
"offer_srtp": true,
"require_srtp": true,
"dial_verbatim": false,
"session_id": "527f78a3262dbe8b0d0a678a9e98ac29"
}'
The smallest request that works — the gateway creates the room and infers SRTP
policy from the trunk’s transport:
curl -X POST http://148.113.58.51:8082/sip/originate \
-H "Authorization: Bearer $ULAI_API_KEY" \
-H 'Content-Type: application/json' \
-d '{
"to_number": "+919363399639",
"trunk_id": "74958094-8c76-419f-aa42-d4eac6768ec2",
"project_id":"fa3f23f8-a75d-44a4-84f0-6a1c828f579f"
}'
Request body
| Field | Type | Required | Description |
|---|
to_number | string | yes | Destination, E.164. Normalised before dialling. |
trunk_id | string | yes | Outbound trunk within the project. The resolver turns it into an address, a transport and credentials — none of which are ever sent on the wire. |
project_id | string | unless ULAI_PROJECT_ID is set | Scopes trunk_id. |
session_id | string | no | Dial into a room that already exists. Omit it and the gateway creates one and returns its id. |
from_number | string | no | Caller ID presented to the trunk. |
name | string | no | Display name in the room roster. Defaults to to_number. |
agent_id | string | no | Recorded in session metadata and visible on the control plane’s discovery feed, so an orchestrator can tell which agent the room is waiting for. Unused by the gateway otherwise. |
max_participants | int | no | Sizes a newly created room. Ignored when session_id is given. Defaults to ULAI_MAX_PARTICIPANTS. |
offer_srtp | bool | no | Offer RTP/SAVP with an a=crypto line. Defaults to true on a TLS trunk, false otherwise. |
require_srtp | bool | no | Fail the call if the answer carries no crypto. Implies offer_srtp. |
dial_verbatim | bool | no | Send to_number without a leading +. For self-hosted or CUSTOM trunks whose dialplan matches literal digit patterns; hosted ITSPs want E.164-with-plus. |
Bodies are capped at 8 KiB. Validation reports every problem at once, so a
dialler does not get one complaint per round trip.
202 Accepted
{
"status": "originating",
"to_number": "+919363399639",
"session_id": "527f78a3262dbe8b0d0a678a9e98ac29",
"created": false,
"name": "sip-module",
"trunk": "Main SBC"
}
created says whether the gateway made the room (and is therefore responsible
for tearing it down). The response returns as soon as the trunk is resolved and
the room exists — the phone is still ringing. Point an agent or a browser at
session_id while it does; the gateway itself joins the room only once the
callee answers.
Errors
| Status | Cause |
|---|
400 | Malformed JSON, failed validation, or no project_id and no ULAI_PROJECT_ID |
401 | Missing or malformed Authorization: Bearer |
404 | No such trunk for that project, or no SIP config for the project |
502 | Room creation failed, or the stored trunk address is malformed |
503 | The gateway is shutting down |
{ "error": "to_number is required\ntrunk_id is required" }
A 404 means the request asked for something that is not there; a 502 means
an upstream failed. Only one of those is worth retrying.
What happens next
Nothing else comes back over HTTP. The call’s progress shows up in three places:
- The log, prefixed with the number.
GET /calls, once the bridge starts.- Telecom events —
CALL_ANSWERED then CALL_HANGUP, published to the
platform’s stream. See routing.
If the callee never answers, the room created for the call is terminated rather
than left running.
No pacing
The gateway dials on every POST. It has no origination rate limit of its own,
so a burst of 429s from a trunk means the caller upstream has to slow down.
1.6 - Operations
Deploying it, networking it, watching it, and fixing it.
- Deployment — container, compose, and
the CI pipeline.
- Networking — ports, host networking,
and why port publishing breaks calls.
- Observability — logs, health
checks, and the control-plane wire log.
- Troubleshooting — symptom first,
cause second.
1.6.1 - Deployment
Running the image on a host.
The gateway is distributed as a container image. A deployment is one container
on a host with a routable public IP, run with host networking.
docker run
docker run -d --name ulai-sip \
-e ULAI_CONTROL_PLANE_URL=https://stgcp.ulai.co.in \
-e SIP_ROUTING_REDIS_URL='redis://USER:PASSWORD@redis.ulai.co.in:6379' \
-e SIP_PUBLIC_IP=148.113.58.51 \
-e SIP_LISTEN_ADDR=0.0.0.0:5060 \
-e SIP_TRANSPORT=udp \
-e SIP_RTP_HOST=0.0.0.0 \
-e SIP_RTP_PORT_LOW=10000 \
-e SIP_RTP_PORT_HIGH=10500 \
-e SIP_HTTP_PORT=8082 \
--network host \
asia-south1-docker.pkg.dev/arctic-operand-415316/ulai/sip:v6
Three things in that command are load-bearing:
--network host — see networking.
Publishing the RTP range with -p spawns hundreds of userland proxies and
rewrites the INVITE’s source address, which the trunk ACL then rejects with
403.SIP_PUBLIC_IP — the address the carrier is told to send media to. It must
be the host’s own routable address, and never 0.0.0.0.SIP_RTP_HOST=0.0.0.0 — what the RTP sockets bind to. On a cloud VM the
public IP usually is not on any interface, so binding to it fails outright.
Pass secrets with --env-file rather than -e if the host’s process list is
readable:
docker run -d --name ulai-sip --network host \
--env-file /etc/ulai/sip.env \
asia-south1-docker.pkg.dev/arctic-operand-415316/ulai/sip:v6
docker compose
services:
sip-gateway:
image: asia-south1-docker.pkg.dev/arctic-operand-415316/ulai/sip:v6
container_name: ulai-sip
network_mode: host
env_file:
- .env
restart: unless-stopped
healthcheck:
test: ["CMD", "./health-check.sh"]
interval: 30s
timeout: 5s
start_period: 15s
retries: 3
network_mode: host replaces a ports: block — the container binds the host’s
ports directly, which is the only arrangement that keeps the carrier’s source
address intact and the RTP range usable.
docker compose up -d
docker compose logs -f
What the image contains
A Debian slim runtime with the gateway binary, libopus, ca-certificates,
and curl plus sipsak for the health check. It runs as an unprivileged user
(sipgw, uid 10001) — SIP 5060 and the RTP range are unprivileged ports, so
nothing here needs root.
Exposed ports:
| Port | Protocol | Purpose |
|---|
8082 | TCP | HTTP API |
5060 | UDP, TCP | SIP signalling |
10000–10500 | UDP | RTP media |
Upgrading
Pull the new tag, then replace the container. There is no state on disk — every
call is in memory and every routing decision is read fresh — so a replacement is
a restart, not a migration:
docker pull asia-south1-docker.pkg.dev/arctic-operand-415316/ulai/sip:v7
docker stop --time 70 ulai-sip
docker rm ulai-sip
docker run -d --name ulai-sip --network host --env-file /etc/ulai/sip.env \
asia-south1-docker.pkg.dev/arctic-operand-415316/ulai/sip:v7
Live calls do not survive the replacement beyond the drain window below, so
prefer a quiet period — or run a second host and move the trunk across.
Restart and shutdown
SIGINT/SIGTERM starts a drain rather than a kill:
shutdown signal received — draining for up to 1m0s
New calls are refused (503 on both legs), in-flight calls get up to 60
seconds to finish, and anything still up is then cancelled with a further 5
seconds of grace. The routing-store connection is closed last, so a draining
call’s CALL_HANGUP still has somewhere to go.
Give the container a matching stop timeout so Docker does not SIGKILL through
the drain:
docker stop --time 70 ulai-sip
1.6.2 - Networking
Ports, host networking, and why port publishing breaks calls.
Ports
| Port | Protocol | Purpose |
|---|
8082 | TCP | HTTP API — /health, /calls, /sip/originate |
5060 | UDP and/or TCP | SIP signalling (5061 conventionally for TLS) |
10000–10500 | UDP | RTP media, one even port per call |
The RTP range sizes the gateway’s concurrency: roughly 250 simultaneous calls at
the default range, since allocation uses even ports only by RTCP convention.
Widen SIP_RTP_PORT_LOW/HIGH to carry more.
The carrier must be able to reach SIP_PUBLIC_IP on the SIP port and the
whole RTP range. A firewall that allows 5060 but not 10000–10500 produces a call
that connects and then has no audio — the single most common misconfiguration.
Port publishing versus host networking
Run the container with --network host.
Publishing the ports with -p breaks the gateway in two separate ways:
- 501 userland proxies.
-p 10000-10500:10000-10500/udp spawns one
docker-proxy process per port. That is slow to start, heavy at rest, and
adds a hop to every media packet. - The trunk ACL rejects the call. A proxied INVITE arrives from the bridge
gateway’s address, not the carrier’s. The resolver checks the source IP
against the trunk’s ACL, does not find it, and answers
403 Forbidden.
The same reasoning is why SIP_PUBLIC_IP must be a routable address: it is
copied into every SDP and Contact the gateway sends, and it is where the
carrier will stream media.
Bind address versus advertised address
Two different variables, routinely confused:
| Variable | Meaning |
|---|
SIP_RTP_HOST (default 0.0.0.0) | What RTP sockets bind to, inside the machine |
SIP_PUBLIC_IP (required) | What the gateway advertises to the carrier |
On AWS, GCP, Azure and most VPS providers the public IP is attached at a NAT or
load-balancer layer and is not present on any interface inside the VM. Binding
to it fails with cannot assign requested address, so the default 0.0.0.0 is
the right answer nearly everywhere.
NAT and ICE
The room side is WebRTC, so it needs ICE. The gateway is STUN-only by default
(stun:stun.l.google.com:19302), which works on a public host and fails behind
symmetric NAT. Add TURN there:
-e TURN_URLS='turn:turn.example.com:3478?transport=udp,turns:turn.example.com:5349' \
-e TURN_USERNAME=ulai \
-e TURN_PASSWORD=...
The startup line reports how many ICE servers were configured:
sip-sfu-gateway: HTTP listening on :8082 (control plane https://stgcp.ulai.co.in, ice_servers=1)
ice_servers=1 means STUN only.
Symmetric RTP
Carriers behind NAT routinely send RTP from a port they never advertised in
SDP. The gateway latches onto the source of the first well-formed packet and
sends there, rather than trusting the SDP address — without which roughly half
of inbound calls would be one-way. See
media.
Transport choice
SIP_TRANSPORT | Notes |
|---|
udp | The default, and what most trunks use |
tcp | Useful where large INVITEs fragment |
tls | Requires SIP_TLS_CERT_PATH and SIP_TLS_KEY_PATH; implies offering SRTP on outbound trunks whose transport is also TLS |
A TLS trunk calling back a UDP-only listener is a real failure mode: the BYE
never arrives and the call would hang forever, which is what
SIP_RTP_TIMEOUT_SECONDS exists to catch.
1.6.3 - Observability
Logs, health checks, and the control-plane wire log.
Reading the log
Every line of a call is prefixed with the number and the leg it arrived on, so
a busy gateway’s log can be read one call at a time:
[+919363399639] [sip-originate] call answered — call_id=3e114ad4…, joining room 527f78a3…
[+919363399639] [sfugw] webrtc state: connected
[+919363399639] [sfugw] DTMF: 5
[+919363399639] [sip-originate] call finished (session 527f78a3…, call_id 3e114ad4…)
| Prefix | Emitted by |
|---|
[sip-gw] | The inbound path |
[sip-originate] | The outbound path |
[sfugw] | The room/WebRTC bridge |
[control-plane] | The control-plane wire log |
[runtime] | Startup runtime tuning |
grep on the number gives one call; grep on the sip_call_id joins the
gateway’s log to carrier CDRs and to the published telecom events.
Health checks
There are two legs, and /health only covers one:
curl -fsS http://127.0.0.1:8082/health
sipsak -s sip:healthcheck@127.0.0.1:5060
health-check.sh in the repo runs both and exits non-zero if either fails.
It is what docker-compose.yml and the CI verification step use. Only one
HEALTHCHECK can be active per image, which is why both checks live in one
script.
/health returns 200 whenever the process is serving. It does not assert
that the routing store is reachable — that is reported at startup and again on
every call that fails because of it.
Live calls
watch -n2 'curl -s localhost:8082/calls | jq'
The list is sorted oldest-first, so rows do not reshuffle between polls. It is
also the number the drain reports:
drain deadline reached with 3 call(s) still up — cancelling them
The control-plane wire log
Every request the gateway makes to the control plane — create, join, terminate
— is logged with its headers and body:
[control-plane] → POST https://stgcp.ulai.co.in/api/v1/sessions headers={Authorization: Bearer ulai…kzM (51 chars), Content-Type: application/json, X-System-Secret: ahd8…3dg (26 chars)} body={"max_participants":8,…}
[control-plane] ← POST /api/v1/sessions 201 Created (128ms)
It exists because “what did we actually send?” is a question best answered by
the log, not by reading the SDK — which matters most when the answer is a 403
and the question is which header was missing.
Three rules it follows:
- Credentials are masked, never printed whole: first and last four
characters plus the length. Enough to tell which secret went out without
putting it in every log shipper that reads this process’s output.
- Auth headers are always reported, present or not. For an auth failure,
Authorization: <not sent> is the single most useful thing a log can say. - Only failure bodies are logged. A success body carries the join ticket and
events token, which are credentials in their own right; a failure body is the
control plane’s explanation.
Events
CALL_ANSWERED and CALL_HANGUP are published to the platform’s telecom
stream for every answered call, carrying the routing decision, the room, the
numbers and — on hangup — duration_seconds. They are the record to build
dashboards and alerting on; the gateway keeps no history of its own beyond the
live table. See routing.
Startup lines worth alerting on
routing store reachable
WARNING: routing store unreachable (...) — every call will be rejected until it recovers
The gateway deliberately starts either way: a store that is down now may be up
a second from now, and a SIP listener answering an honest 500 is worth more
than one that is not there at all.
1.6.4 - Troubleshooting
Symptom first, cause second.
The gateway will not start
sip-sfu-gateway: ULAI_CONTROL_PLANE_URL is required (e.g. https://stgcp.ulai.co.in)
SIP_PUBLIC_IP must be a routable address, not 0.0.0.0 — it is advertised to the carrier
Configuration is validated once, and every problem is reported together — fix
them all, then restart. 0.0.0.0 in SIP_PUBLIC_IP is rejected on purpose: it
is not a listen address, it is what the carrier is told to stream media to.
Other startup failures:
| Message | Cause |
|---|
SIP_TLS_CERT_PATH and SIP_TLS_KEY_PATH are required when SIP_TRANSPORT=tls | TLS transport with no certificate |
SIP_RTP_PORT_LOW/HIGH (…) must be a valid ascending port range | Inverted or out-of-range ports |
routing service: … | The Redis URL is malformed — this one does fail startup |
SIP runtime init: … | The SIP port is already bound, or the TLS material is unreadable |
Calls are rejected
The SIP status tells you which check failed:
| Status | Meaning | Where to look |
|---|
403 Forbidden | Source IP not in the trunk’s ACL | The carrier’s signalling address, and whether you are behind a Docker port proxy — see networking |
404 Not Found | Unknown SIP domain, unassigned number, or no project config | The routing store: is the domain mapped, is the number assigned |
480 Temporarily Unavailable | Routed, but no dispatcher rule matched | The project’s dispatcher rules |
500 Server Internal Error | The routing lookup itself failed | Redis connectivity — WARNING: routing store unreachable will be in the log |
503 Session unavailable | The control plane would not create a room | The [control-plane] wire log |
503 Shutting down | Drain in progress | Deploy timing |
Every rejection is logged with the reason immediately before it:
[+919363399639] [sip-gw] rejecting 403: source IP not allowed by the trunk ACL: …
The call connects but there is no audio
Almost always the media path, not the signalling path:
- The RTP range is not open. 5060 allowed, 10000–10500 blocked, is the
classic. Check the firewall and the security group.
SIP_PUBLIC_IP is wrong. The carrier is streaming to whatever the SDP
said. Check it against the host’s actual routable address.- Port publishing instead of host networking. Media arrives at the proxy
rather than the process.
- One-way only? That is usually NAT on the carrier’s side. The gateway
already latches onto the observed source address; if it is still one-way,
the outbound direction is being dropped upstream.
Outbound calls fail
| Response | Cause |
|---|
401 | No Authorization: Bearer header, or a malformed one |
400 project_id is required (or set ULAI_PROJECT_ID) | Neither the body nor the environment names a project |
404 | That trunk_id does not exist in that project |
502 | Room creation failed, or the stored trunk address is malformed |
If the 202 comes back but the phone never rings, the failure is in the
background half and only the log has it:
[+919363399639] [sip-originate] Originate failed: …
[+919363399639] [sip-originate] RATE LIMITED by trunk … (SIP 429, Retry-After 30s) — this gateway does not pace originations; the caller must slow down
The room created for that call is terminated rather than left running.
A carrier with secure trunking enabled answered a plain RTP/AVP offer. The
gateway infers SRTP from the trunk’s transport — TLS trunks offer it, UDP and
TCP do not — so either fix the stored transport, or force it per request with
"offer_srtp": true. See
routing.
404 from the trunk on a self-hosted SBC
A dialplan matching literal digit patterns will not match a leading +. Send
"dial_verbatim": true. Hosted ITSPs want the opposite — E.164 with the plus.
A call never ends
If the far end’s BYE is lost — a TLS trunk calling back a UDP-only listener
never reaches us at all — the call and its room would stay up until the process
exits. SIP_RTP_TIMEOUT_SECONDS (default 30) is the backstop: a call whose
inbound audio stops for that long is ended as if the far end had hung up. It is
paused while the call is on hold. Setting it to 0 disables the backstop
entirely, which is rarely what you want.
The agent talks to a ringtone
It should not — the gateway joins the room only after the callee answers,
precisely so a participant appearing in the roster is a reliable cue. If it
happens, something else joined the room early; the gateway’s own join is logged
as call answered — … joining room ….
The agent keeps talking after the caller hangs up
The session is not being terminated with the call. A room the gateway created
is always torn down; a room it was pointed at (X-Session-Id, or session_id
on originate) is torn down too unless
SIP_TERMINATE_SESSION_ON_HANGUP=false. Check that variable first.
Duplicate agents on one call
A mid-call re-INVITE handled as a new call produces exactly this — a second
agent, a second model session, a second billing row. In-dialog INVITEs are
handled separately for that reason. If you see it, capture the SIP flow and
check whether the second invocation followed a re-INVITE from the carrier’s SBC.
2 - Agent Server
The voice engine, shipped as a container image.
ulai-agent-server is the voice engine. It joins an Ulai SFU room, connects an
AI backend, and runs the conversation — the audio chain, turn detection,
barge-in, the silence ladder and the end-call contract.
It ships as a container image. There is nothing to build and nothing to
compile.
It holds no database, no agent store and no credentials. Everything about how
a call runs — the prompt, the voice, the GCP project, the service-account key
— arrives from whatever drives it, per call. That is why the configuration
below is so short.
Run it
docker run -d --name agent_server --restart unless-stopped \
-p 127.0.0.1:50052:50052 \
-p 8000:8000 \
-e APP_GRPC_LISTEN_PORT=50052 \
-e APP_HTTP_PORT=8000 \
-e ULAI_GRPC_API_KEY='<64-character key>' \
asia-south1-docker.pkg.dev/arctic-operand-415316/ulai/agent_server:v1.02
Check it:
curl -s http://localhost:8000/health # → ok
That is the whole installation. See
deploy it for the full walk-through,
including the part that actually makes calls happen.
What you still need
The agent server does not place calls by itself. It waits on port 50052 for
an agent client to tell it which room to join and which agent to run.
agent client ──gRPC :50052──▶ agent server ──▶ Ulai SFU room
(drives calls) (this image) (the conversation)
Both sides must share the same key: whatever you set as ULAI_GRPC_API_KEY
here, the client must present. Nothing happens until a client connects.
In this section
- Overview — what it does, what it
deliberately does not, and where it sits.
- Deploy it — pull, run, verify, and
connect a client.
- Configuration — every environment
variable, and the longer list of things that are not environment.
- What a call carries — why the environment
is short, and what arrives per call instead.
- Interfaces — the two ports and what talks
to them.
- Operations — deployment, observability
and troubleshooting.
2.1 - Overview
What the engine does, what it refuses to do, and where it sits.
What it is
The Ulai Agent Server is a single Go binary (orchestrator) that runs one side
of a voice conversation. It joins an Ulai SFU room as an ordinary participant,
connects an AI backend to that room, and manages the conversation until
somebody ends it.
Everything it owns is timing-critical:
- The audio chain — pre-clean, denoise, near-field foreground isolation,
neural VAD.
- Turn detection — who has the floor, and when that changes.
- Barge-in — cutting the agent mid-sentence when the caller speaks, and
accounting for how much of the turn was actually heard.
- The silence ladder — nudges when a caller goes quiet, and a hangup when
they stay quiet.
- The end-call contract — the evidence a model must produce before it is
allowed to hang up on a human.
What it is not
It is not a place where anything is stored. There is no database, no agent
table, no recording bucket and no credential file. A call’s configuration
arrives in the request that starts it, and everything worth keeping leaves on
the event stream for somebody else to write down.
That is a deliberate constraint rather than an omission. It is what lets one
engine serve agents belonging to different customers, billed to different GCP
projects, running on different AI vendors — and it is what makes the binary
safe to hand to somebody else.
| It owns | Somebody else owns |
|---|
| Audio, turns, barge-in, playback timing | Agent configuration and where it is stored |
| The AI session and its lifecycle | Which GCP project and key that session uses |
| Tool dispatch and the end-call contract | What a custom tool actually does |
| Emitting call events | Writing call records, costs and transcripts |
Where it sits
agent client ──gRPC──▶ agent server ──WebSocket──▶ Ulai SFU room
(your logic) (this) (the conversation)
│ │
│ └──▶ AI backend (Gemini Live, …)
│
└──▶ your database, your dashboards, your billing
The agent client decides which agent runs, what it says, and where it is
billed. It sends all of that with each call. The agent server runs the
call. The two speak over port 50052 — see interfaces.
A client that crashes mid-call does not drop the caller. The engine carries on
with the configuration it was given; what is lost is the client’s ability to
steer and to record, not the conversation.
How it is shipped
As a container image, and only as one. There is no source build, no package to
install and no plugin to load — everything the engine needs is inside it,
including the compiled audio libraries.
What you supply is a port, a key, and a client to drive it. See
deploy it.
What it does not carry
Audio never leaves the media path. The agent client receives transcripts and
sends text; no audio frame crosses the control port.
That is what keeps a slow or distant client from being heard: the round trip it
adds lands between turns, where tens of milliseconds are invisible — not
inside the frame cadence, where they are a stutter the caller hears.
2.2 - Deploy it
Pull, run, verify, and connect a client.
Before you start
You need:
- Docker, and access to the registry the image lives in.
- A key for port 50052 — any 64-character random string. The agent client
must present the same one.
- An agent client to drive it. The agent server never places a call by
itself; it waits to be told which room to join.
You do not need a GCP project, a service-account key, a model name or a
prompt on this host. All of that arrives with each call.
1. Pull
docker login asia-south1-docker.pkg.dev
docker pull asia-south1-docker.pkg.dev/arctic-operand-415316/ulai/agent_server:v1.02
Pin the tag. latest makes it impossible to say afterwards what was running.
2. Run
docker run -d --name agent_server --restart unless-stopped \
-p 127.0.0.1:50052:50052 \
-p 8000:8000 \
-e APP_GRPC_LISTEN_PORT=50052 \
-e APP_HTTP_PORT=8000 \
-e ULAI_GRPC_API_KEY='<64-character key>' \
asia-south1-docker.pkg.dev/arctic-operand-415316/ulai/agent_server:v1.02
Why 127.0.0.1 on 50052
The gRPC port has no encryption of its own. The key — and the credentials
each call carries — cross it in clear.
Bind it to loopback and put a TLS proxy in front, or keep it on a private
network the client shares. Do not publish it to the internet.
With docker compose
services:
agent_server:
image: asia-south1-docker.pkg.dev/arctic-operand-415316/ulai/agent_server:v1.02
restart: unless-stopped
ports:
- "127.0.0.1:50052:50052"
- "8000:8000"
environment:
APP_GRPC_LISTEN_PORT: 50052
APP_HTTP_PORT: 8000
ULAI_GRPC_API_KEY: "${ULAI_GRPC_API_KEY}"
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
interval: 30s
timeout: 5s
retries: 3
start_period: 10s
3. Verify it started
curl -s http://localhost:8000/health
ok means the process is alive. It says nothing about calls — the server is
healthy with zero calls running, which is its normal state.
Now read back what it thinks its configuration is:
docker logs agent_server 2>&1 | grep -A 25 "agent server configuration"
[env] ---- agent server configuration ----
[env] APP_GRPC_LISTEN_PORT 50052 # gRPC port, all interfaces
[env] APP_HTTP_PORT 8000 # health only
[env] ULAI_GRPC_API_KEY set len=64 sha256:57a4cbf1 # demanded of callers; unset = NO AUTH
[env] ONNXRUNTIME_LIB /usr/local/lib/libonnxruntime.so (file present) # Smart Turn and Silero; the image sets it
[env] ---- end agent server configuration ----
[grpc] agentsession.v1 listening on [::]:50052 (AgentBridge served; AgentSession/AgentDispatch served)
Four things to confirm:
| Check | Why |
|---|
ULAI_GRPC_API_KEY set len=64 | If it says (unset), anyone who reaches the port can control calls. |
ONNXRUNTIME_LIB … (file present) | The turn and voice-activity models cascade calls use are in place. |
No NOT READ BY THIS PROCESS line | Anything listed there is set on this host and ignored — usually a rename, or a setting that now travels with the call. |
[grpc] … listening | The control port is up. |
4. Point a client at it
Configure the agent client with this host’s address and the same key. In a
typical client that is:
| Client setting | Value |
|---|
| Orchestrator address | host:50052 |
| Orchestrator API key | the same ULAI_GRPC_API_KEY |
| TLS | on, if you put a proxy in front |
Nothing appears in the agent server’s log until the client connects and starts
a call. Silence here is normal.
5. Confirm a real call
Place one call through the client, then:
docker logs agent_server --tail 50
A healthy call looks like this:
[bridge] join: bridge=48c7ad… agent_id=d1a6b0… participant=agent_7f85…
[AIRoute] ai_project=my-project location=us-central1 creds=supplied (2347 bytes) (from the agent profile)
[bridge:48c7ad…] using the profile supplied with the request — no agent lookup, no database
[gemini] connected, greeting deferred (call=… model=… voice=Kore)
[bridge:48c7ad…] greeting primed
Two lines are the ones to know:
creds=supplied (N bytes) — the client sent credentials. If it says
creds=engine default, the client sent none and the call will be refused.greeting primed — the backend connected and the agent is about to
speak. If this never appears, the call died before it could talk.
Upgrading
docker pull …/agent_server:v1.03
docker stop agent_server && docker rm agent_server
docker run -d … …/agent_server:v1.03 # same flags
Calls in flight during a restart end — there is no session migration. Restart
when the log is quiet, or accept the dropped calls.
Next
2.3 - What a call carries
Why the environment is short, and what arrives per call instead.
The agent server is configured by two things, and only one of them is on this
host.
2.3.1 - The agent profile
Everything about how a call runs, sent by the client.
Almost nothing about a conversation is configured on the agent server. The
prompt, the voice, the GCP project, the service-account key, which behaviours
are on — all of it arrives with each call, in something called the agent
profile, sent by the agent client.
This matters to you as an operator for three reasons.
1. It explains the short environment list
You are not missing settings. There is no PROMPT, no VOICE, no
GEMINI_MODEL to set here, because those are properties of an agent, not of
a host. One agent server runs many different agents at once, each with its
own.
2. It means one host can serve many customers
Because the GCP project and the service-account key travel with each call, a
single agent server can run:
- agent A, billed to customer A’s GCP project, on customer A’s key
- agent B, billed to customer B’s, on customer B’s key
at the same time. As host configuration this would be impossible — one
deployment could only ever serve one account, and serving two meant running two
servers.
3. It means this host holds no credentials
There is no service-account key on this machine and no file to mount. If
somebody copies the image, they get no customer’s credentials with it.
The trade is that a call carrying no credentials cannot run. There is
nothing to fall back onto, so it is refused:
gemini: this call carried no credentials for project "my-project" —
the agent server holds none of its own, by design. Set google_credentials_json
on the agent, or GOOGLE_APPLICATION_CREDENTIALS on the agent client that sends
its profile
If you see that, the fix is on the client, not here. See
troubleshooting.
What is in a profile
You do not set these — the client does — but knowing what a call carries makes
the logs readable.
| Group | Examples |
|---|
| The conversation | prompt, greeting, closing line, voice, model, language |
| Placement | GCP project, region, service-account key, which AI vendor |
| Behaviour | can the caller interrupt, is the greeting interruptible, silence handling, answering-machine detection |
| Tools | the functions this agent may call, answered by the client |
Each field falls back on its own. A client sending a project but no region gets
that project in the default region — not a half-configured call.
No environment variable on this host fills in a field the profile leaves out.
The project and the key in particular must come from the call.
2.3.2 - Tools
Why an agent can look things up, and where that work happens.
An agent can do more than talk. Mid-call it can look up a balance, an order
status, an appointment slot — anything the business it belongs to can answer.
None of that work happens on this host. The agent server has no database
and no idea what any of those tools mean. It carries the request out to the
agent client and carries the answer back.
model asks for "check_balance"
│
▼
agent server ──▶ agent client ──▶ the business's own system
◀──────────── answer ◀───────────────┘
│
▼
model continues the conversation
What this means operationally
Tools are configured on the agent, not here. There is nothing to install,
declare or permit on this host to make a tool available.
A slow tool is heard as silence. While the client is answering, the model
is not speaking, and the caller hears that. The wait is bounded and always ends
in an answer to the model — if the client does not answer in time, the model is
told the lookup failed so it can apologise and carry on. A caller hearing “I
couldn’t pull that up just now” is a far better outcome than a line that goes
quiet.
So a tool that is slow on the client side shows up as pauses in conversations,
not as errors in this log.
Two tools belong to the server. end_call — how a model hangs up, subject
to a contract that checks it is entitled to — and a no-op tool declared beside
it. The second exists so that a model which feels the urge to “use a tool” at a
moment that is not an ending has somewhere harmless to put it, rather than
reaching for end_call and cutting a live human off.
It is answered instantly, inside the server, precisely because sending it out
and back would spend a multi-second silence avoiding a mistake that costs
nothing.
In the logs
Tool activity appears against the call’s bridge id. A call that pauses oddly,
with the agent going quiet and then apologising, is usually a tool the client
was slow to answer — the place to look is the client’s logs, not this one.
2.3.3 - The shape of a call
From join to hangup, and who decides what along the way.
Joining
The room exists first. Whoever created it is the only party that knows which
room the other participants are in, so the agent server never creates one — it
is told which to join.
client ──JoinBridge──▶ server ──join──▶ control plane
│
├──▶ SFU room (WebSocket, Opus)
└──▶ AI backend (the placement on the profile)
JoinBridge returns once the agent has a seat. The backend handshake
continues after that.
Who speaks first
On a bridge the agent does. The room is already live, nobody rang, and an agent
that joins and stays silent reads as a broken connection.
opening overrides this. Set it to caller when the agent is joining a
conversation already in progress and should listen rather than announce itself.
During
The engine owns the timing. Every 20 ms of caller audio runs the chain —
pre-clean, denoise, near-field foreground isolation, neural VAD — and the VAD
is the authoritative turn driver.
The agent client receives transcripts and turn events and can steer between
turns. What it cannot do is touch the media path: there is no way to send audio
into the call and no way to change the audio processing. Timing stays with the
engine.
Barge-in
When the caller speaks over the agent, the agent stops. What matters afterwards
is not what the agent generated but what the caller actually heard, so a
turn-ended event carries both:
An agent turn generated in full and heard for 300 ms did not happen, whatever
the transcript says.
A client deciding “have they been told about the fee?” needs the second number,
not the first.
Ending
Three ways a call ends:
| |
|---|
| The caller hangs up | The leg drops; the room reports it. |
| The client ends it | Optionally running the closing sequence first. |
The model calls end_call | Subject to the end-call contract. |
end_call is not a request the model gets for free. It must produce evidence —
which of the legitimate endings this is, whether the closing question was
actually spoken and answered — and that evidence is cross-checked against what
the engine independently observed. A rejected end_call is answered with a
re-prompt, so the agent keeps talking rather than going silent.
Every rejection is a prompt, and the engine is careful that rejections do not
themselves become a source of repetition.
Detaching is not hanging up
If the client process dies mid-call, the engine carries on. The caller is not
dropped because the record-keeper crashed; the cost is a reporting gap, which
is the right one to take.
2.4 - Configuration
The short list that is environment, and the long list that is not.
Configuration is split by a single rule:
If it decides how a call runs, it travels with the call.
If it decides how the process runs, it is an environment variable.
That split is why the environment list below is so short, and why the same
binary can serve agents belonging to different customers at the same time. The
per-call half is documented under
the agent profile.
The process prints its whole configuration at boot, including which variables
are set but not read — usually a rename, or a setting that has moved to the
call:
[env] ---- agent server configuration ----
[env] APP_GRPC_LISTEN_PORT 50052 # gRPC port, all interfaces
[env] ULAI_GRPC_API_KEY set len=64 sha256:57a4cbf1 # demanded of callers; unset = NO AUTH
[env] ONNXRUNTIME_LIB /usr/local/lib/libonnxruntime.so (file present) # Smart Turn and Silero; the image sets it
...
[env] ---- end agent server configuration ----
Process
This is the whole list. Nothing else in the environment is read.
| Variable | Default | What it does |
|---|
APP_GRPC_LISTEN_PORT | 50052 | The gRPC control surface. Binds all interfaces inside the container. |
APP_HTTP_PORT | 8080 | /health, and nothing else. |
ULAI_GRPC_API_KEY | (none) | Required as x-api-key on every RPC. Unset means no authentication. |
PROMPT_LOG | (on) | Logs every text turn and tool result sent to the model. off stops it. Audio is never logged. |
APP_PPROF_PORT | 6060 | pprof. Binds container-loopback, so publishing the port does nothing. |
PROFILER_ENABLED | false | Starts pprof at all. |
ONNXRUNTIME_LIB | (set by the image) | Where ONNX Runtime is, for the turn and voice-activity models cascade calls use. Leave it alone. |
GOMAXPROCS, GOMEMLIMIT, GOGC | (auto) | Runtime tuning. GOMEMLIMIT is derived from the cgroup limit when unset. |
ULAI_GRPC_API_KEY
Leave it unset and the server accepts unauthenticated calls — including ones
that disconnect live agents. It logs a warning saying so, once, at startup.
The gRPC listener has no transport security of its own. The key, and any
credentials a client sends with a call, cross the wire in clear unless TLS is
terminated in front of the port. Publish it to 127.0.0.1 and put a proxy
there.
No longer read
Earlier images also read call settings from the environment. These are now
ignored, because every call setting arrives with the call:
AI_BACKEND, AI_CLASSIFY_BACKEND, AI_TRANSCRIBE_MODEL, ULAI_LIVE_*,
CASCADE_*, AMD_ENABLED, AUDIO_NOTHING, LOST_UTTERANCE_REPLAY,
GEMINI_PROJECT_ID, GEMINI_LIVE_LOCATION, GOOGLE_APPLICATION_CREDENTIALS.
Remove them from your deployment. While they are set, the boot report lists the
AI_*, GEMINI_*, GOOGLE_* and ULAI_* ones on its NOT READ BY THIS PROCESS line.
A call that arrives with no project or no key is refused with a message naming
the field and both places it can be set. It is not quietly run against
whatever ambient identity the host happens to carry — on a GCE instance that
succeeds at authentication and then fails the Live handshake with insufficient authentication scopes, which describes neither the missing setting nor the
account it borrowed.
2.5 - Interfaces
Two ports, and what talks to them.
The image exposes two ports and nothing else. There is no web UI, no admin
endpoint and no configuration API — the agent server is driven entirely by the
agent client.
| Port | Protocol | Who connects | Expose it? |
|---|
50052 | gRPC | the agent client | Loopback or private network only |
8000 | HTTP | your monitoring | As needed |
Port 8000 — health
One endpoint:
curl -s http://localhost:8000/health # → ok
A liveness check, not a readiness one. It returns ok as soon as the
process is up, including when no calls are running — which is the normal state.
It does not tell you whether calls succeed; the log does that.
Safe to expose to a load balancer or monitoring system. It reveals nothing.
Port 50052 — control
This is where the agent client tells the server which room to join, which agent
to run, and where to bill it. Everything about a call travels over this port.
It must be protected. Two reasons:
- There is no encryption. The server holds no TLS certificate. The API key
and the service-account credentials each call carries cross this port in
clear text.
- It controls live calls. A caller who reaches it can place agents into
rooms and disconnect ones already talking.
The safe shapes are:
- Bind to
127.0.0.1 and run a TLS-terminating proxy in front — what the
published examples do. - Keep it on a private network that only the agent client can reach.
Publishing 0.0.0.0:50052 to the internet is not one of them.
Authentication
Every request must present ULAI_GRPC_API_KEY. The client is configured with
the same value.
Leave the variable unset and the server accepts everything, including
requests that disconnect live agents. It logs a warning saying so at startup,
once:
[grpc] WARNING: ULAI_GRPC_API_KEY is unset — :50052 accepts unauthenticated
calls, including ones that disconnect live agents
One key, shared by every client. There is no per-client identity and no way to
revoke one caller without rotating for all of them, so treat the key as an
infrastructure secret rather than a per-team credential.
What it actually serves
For completeness, the port serves three gRPC services — AgentBridge (put an
agent in a room, take it out), AgentSession (drive a live conversation) and
AgentDispatch (receive calls as they start). Operating the server requires no
knowledge of them; they matter to whoever builds the client.
Server reflection is enabled, so grpcurl can inspect the port for debugging —
behind the same key:
grpcurl -H 'x-api-key: <key>' -plaintext localhost:50052 list
Outbound connections
The server also makes connections out, which your egress rules must allow:
| To | For |
|---|
| The Ulai control plane and SFU nodes | joining rooms and carrying audio |
| The AI backend (Vertex AI by default; also Cloud Speech-to-Text and Text-to-Speech on cascade calls) | the conversation itself |
Both are named per call by the client, not configured here.
2.6 - Operations
Deployment, observability and troubleshooting.
Running it, and finding out why it is not running.
2.6.1 - Deployment
The image, the ports, and the order things must be deployed in.
The image
It is a heavy image and has to be: it owns the audio. RNNoise is compiled
from vendored C, TEN VAD is a prebuilt .so, and libopus, libsoxr and
libspeexdsp are linked for the transport and the DSP chain. ONNX Runtime is
included for the turn and voice-activity models that cascade calls use, and
ONNXRUNTIME_LIB points at it.
The runtime stage is Debian slim and carries only the binary plus those shared
libraries. It owns no business data — no database, no agent store, no
credentials.
Ports
| Port | What | Expose it? |
|---|
50052 | gRPC (agentsession.v1) | Loopback or private network only. |
8000 | /health | As needed. |
6060 | pprof | Binds container-loopback; publishing it does nothing. |
There is no TLS
The gRPC server has no transport credentials of its own. The API key and any
service-account key a client sends cross the wire in clear.
Publish 50052 to 127.0.0.1 and terminate TLS in front of it. The SDK’s
Secure() expects exactly that.
Health
GET /health returns ok. It is a liveness check, not a readiness one — the
process is healthy with zero calls running, which is the normal state.
Deployment order
The engine holds no project and no credentials, so the client that sends them
must be deployed first. Deploying the server ahead of the client means every
call is refused for want of a key.
The order:
- Deploy the client with its placement configured.
- Confirm
[AIRoute] … creds=supplied (N bytes) on a call. - Deploy the server.
Reversing it produces a working-looking server and a fleet of refused calls.
Sizing
One process handles many concurrent calls; the limit is CPU, and the audio
chain is what spends it. Each call runs denoise, foreground isolation and
neural VAD every 20 ms, and a cascade call runs its turn model on top.
GOMEMLIMIT is derived from the cgroup limit when unset — the boot log says
what it picked:
[runtime] GOMEMLIMIT=13589MiB (/proc/meminfo limit=15987MiB, 85%)
Rolling restarts
Stop drains in-flight RPCs and then escalates to a hard stop after a timeout.
One long-lived session stream must not hold a deploy open indefinitely — a
session stream ends when its call does, which can be minutes away.
Calls in flight during a restart end. There is no session migration.
2.6.2 - Observability
The log lines worth knowing, and what they tell you.
Everything goes to stdout, so docker logs is the whole interface. Every line
for a call carries its bridge id, which makes one call one filter:
docker logs agent_server --tail 100
docker logs agent_server 2>&1 | grep 48c7ad0bc30f # one call
docker logs -f agent_server # follow
There are no metrics endpoints. Ship stdout to wherever you keep logs.
At boot
The whole configuration is printed, including variables that are set but not
read — usually a rename, or a setting that now travels with the call:
[env] ---- agent server configuration ----
[env] APP_GRPC_LISTEN_PORT 50052 # gRPC port, all interfaces
[env] ULAI_GRPC_API_KEY set len=64 sha256:57a4cbf1 # demanded of callers; unset = NO AUTH
[env] ONNXRUNTIME_LIB /usr/local/lib/libonnxruntime.so (file present) # Smart Turn and Silero; the image sets it
[env] NOT READ BY THIS PROCESS: AI_BACKEND APP_ENV APP_NAME SERVICE_HTTP_PORT
[env] (set in the environment but nothing here looks at them — usually a rename)
[env] ---- end agent server configuration ----
Secrets are shown as a length and a hash, never a value. A path that names a
missing file says !! NO SUCH FILE.
Also at boot:
[grpc] agentsession.v1 listening on [::]:50052 (AgentBridge served; AgentSession/AgentDispatch served)
Per call
| Line | Means |
|---|
[bridge] join: bridge=… agent_id=… participant=… | The agent has a seat. |
[AIRoute] ai_project=… creds=supplied (N bytes) (from the agent profile) | The profile carried a key. The one to check first. |
using the profile supplied with the request | No database lookup happened. |
[gemini] connected, greeting deferred | The backend handshake succeeded. |
ready — mode=… llm=…@… tts=…@… … | A cascade call is up, with every setting it is running on. |
greeting primed | The agent is about to speak. |
roster: N other participant(s) in room | Who else is there. |
[AIRoute] never prints a credential. It shows creds=supplied (2347 bytes)
or creds=engine default — the latter meaning the profile carried none, which
is a refusal.
Audio
When a call’s profile sets audio_nothing, the chain is cut down to the
denoiser and the call’s log says so loudly:
[AUDIO] audio_nothing — pure RNNoise (preprocessor + foreground gate disabled)
That line in production is almost always a mistake — see
troubleshooting.
Prompts
PROMPT_LOG is on unless set to off. Every text turn and tool result sent to
the model is logged as an [→AI] line — the persona, the nudges, the closing
instruction — which is the only record of what the model was actually told.
Turn it off if prompts or caller details should not reach your log store.
Audio is never logged either way.
2.6.3 - Troubleshooting
Symptoms, and what actually causes them.
Start here
docker ps --filter name=agent_server # is it running?
curl -s http://localhost:8000/health # → ok
docker logs agent_server --tail 100 # what happened
Every line for a call carries its bridge id, so one call is one filter:
docker logs agent_server 2>&1 | grep 48c7ad0bc30f
Look for backend connect failed: right after joined session=….
this call carried no credentials for project "…"
The profile carried a project but no key. The engine holds none of its own, so
there is nothing to fall back onto.
Confirm it in this server’s log — creds=engine default on the [AIRoute]
line means the call carried no key:
docker logs agent_server 2>&1 | grep AIRoute | tail -5
The fix is on the agent client, which is the thing that must send a key.
Nothing you change on this host will help: it holds no credentials by design.
this call carried no AI project
Same shape, other field. The profile named neither a project nor a key.
insufficient authentication scopes
Seen on older builds that fell back to ambient credentials. The host’s own
identity — a GCE instance service account — authenticated successfully and then
failed the Live handshake, because it lacks the cloud-platform scope Vertex
needs.
Current images refuse the call before this can happen, with a message naming
the missing field. If you see this, you are on an older image — check the tag
with docker inspect --format '{{.Config.Image}}' agent_server.
DetectAnsweringMachine needs a Classifier
A call asked for answering-machine detection on an image that could not supply
a classifier for it. Current images turn detection off rather than failing the
call. Upgrade the image, or have the client stop asking for AMD.
The agent joins but never speaks
Check for greeting primed. If it is absent, the backend never connected —
see above. If it is present and there is still silence, the caller’s side is
the place to look: the room roster line says whether anyone else is actually
there.
A setting on this host has no effect
You set AI_BACKEND, AMD_ENABLED, AUDIO_NOTHING or similar here, and calls
behave as if you had not. Those are no longer read from the environment — see
no longer read.
ONNXRUNTIME_LIB … !! NO SUCH FILE at boot
The image sets this to its own copy of ONNX Runtime, so this means it was
overridden. Cascade calls still run, but without the turn and voice-activity
models: they fall back to the built-in VAD and end turns on silence alone.
Remove the override.
Calls are refused with unauthorized
The caller’s key does not match ULAI_GRPC_API_KEY. The boot log shows a hash
of what the server expects:
[env] ULAI_GRPC_API_KEY set len=64 sha256:57a4cbf1
If it says (unset), the server is accepting everything, and the failure
is elsewhere.
Tools are configured on the agent and answered by the agent client — there is
nothing to enable on this host. A tool that never runs is a client-side
problem.
A call that pauses, goes quiet and then apologises is usually a tool the client
answered too slowly. See tools.
Captions stop after the first utterance
Every caller utterance landing in one caption is a segment that never closed.
Fixed in current images; on an older one the caller’s transcript accumulates
into a single bubble while the agent’s lines appear separately. Upgrade.
The caller sounds gated or clipped
Have the client send audio_nothing on one call. It strips the chain to the
denoiser alone, so if the problem disappears it is the pre-clean stage or the
near-field foreground gate. There is no host-wide switch for this any more.
It is a diagnostic, not a setting. The full chain is the tuned one, and the
stage it removes is what keeps a noisy room out of the model.