This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Documentation

Ulai is built from independent modules. Each one is documented on its own, with its own configuration, API and operational notes.

Modules

More modules are documented here as they land.

1 - SIP Gateway

SIP trunks in, Ulai SFU voice rooms out.

ulai-sip-module is a single Go binary (sip-sfu-gateway) that puts a phone call into an Ulai SFU room. A carrier’s INVITE is routed, admitted and answered; an API call dials out through a trunk. Either way the caller ends up as an ordinary WebRTC participant talking to whoever else is in the room.

Source: github.com/ulaidotin/ulai-sip-module

Run it

docker run -d --name ulai-sip \
  -e ULAI_CONTROL_PLANE_URL=https://stgcp.ulai.co.in \
  -e SIP_ROUTING_REDIS_URL='redis://USER:PASSWORD@redis.ulai.co.in:6379' \
  -e SIP_PUBLIC_IP=148.113.58.51 \
  -e SIP_LISTEN_ADDR=0.0.0.0:5060 \
  -e SIP_TRANSPORT=udp \
  -e SIP_RTP_HOST=0.0.0.0 \
  -e SIP_RTP_PORT_LOW=10000 \
  -e SIP_RTP_PORT_HIGH=10500 \
  -e SIP_HTTP_PORT=8082 \
  --network host \
  asia-south1-docker.pkg.dev/arctic-operand-415316/ulai/sip:v6

--network host is not optional — see networking.

Dial out

curl -X POST http://148.113.58.51:8082/sip/originate \
  -H 'Authorization: Bearer ulai_live_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' \
  -H 'Content-Type: application/json' \
  -d '{
    "to_number":        "+919363399639",
    "trunk_id":         "74958094-8c76-419f-aa42-d4eac6768ec2",
    "project_id":       "fa3f23f8-a75d-44a4-84f0-6a1c828f579f",
    "from_number":      "+13187184515",
    "name":             "sip-module",
    "max_participants": 8,
    "offer_srtp":       true,
    "require_srtp":     true,
    "dial_verbatim":    false,
    "session_id":       "527f78a3262dbe8b0d0a678a9e98ac29"
  }'

In this section

  • Overview — what it does, what it does not do, and where it sits in the platform.
  • Getting started — prerequisites, build, configuration, and your first inbound and outbound call.
  • Concepts — call flows, the routing plane, and the media pipeline.
  • Configuration — every environment variable and built-in timeout.
  • HTTP API — /health, /calls and /sip/originate.
  • Operations — deployment, networking, observability and troubleshooting.

1.1 - Overview

What the gateway does, and where it sits in the platform.

What it is

The Ulai SIP Gateway is a single Go binary (sip-sfu-gateway) that joins two worlds:

  • SIP/RTP — a carrier trunk, speaking SIP over UDP, TCP or TLS, carrying G.711 audio over RTP or SRTP.
  • Ulai SFU rooms — WebRTC sessions on the Ulai control plane, where browsers and AI agents meet.

A phone call that reaches the gateway becomes an ordinary participant in a room. Everything the gateway does is in service of that one sentence: decide whether the call is allowed, get a room for it, answer it, and move audio in both directions until somebody hangs up.

It handles both directions of origination:

DirectionTriggerThe gateway is
InboundA carrier sends an INVITE to the SIP listenerthe answering party (UAS)
OutboundPOST /sip/originate with a number and a trunk idthe calling party (UAC)

What it is not

  • Not an agent. The gateway never talks to a model, never decides what to say, and has no notion of a conversation. It publishes CALL_ANSWERED and lets the platform’s dispatcher summon whatever should join the room.
  • Not a media processor. No noise suppression, no VAD, no recording, no playback accounting. It transcodes and it forwards.
  • Not a rate limiter. It dials on every POST /sip/originate. If a trunk answers 429, the caller upstream is the one that has to slow down.
  • Not a routing database. Numbers, trunks, ACLs and dispatcher rules live in the platform’s routing store; the gateway only reads them.

Where it sits

   carrier / ITSP                gateway                  Ulai platform
  ┌───────────────┐        ┌──────────────────┐        ┌──────────────────┐
  │ SIP trunk     │──────▶ │ SIP listener     │        │ control plane    │
  │ (udp/tcp/tls) │ INVITE │  :5060           │──────▶ │  create / join / │
  │               │        │                  │  HTTPS │  terminate       │
  │               │◀─────  │ RTP :10000-10500 │        └──────────────────┘
  └───────────────┘  RTP   │                  │        ┌──────────────────┐
                           │ HTTP API :8082   │──────▶ │ routing store    │
                           │  /sip/originate  │  Redis │  (sip-resolver)  │
                           │  /calls /health  │        └──────────────────┘
                           │                  │        ┌──────────────────┐
                           │ WebRTC / Opus    │──────▶ │ SFU room         │
                           └──────────────────┘  SRTP  │  agent, browsers │
                                                       └──────────────────┘

Three external dependencies, all required:

  1. The control plane (ULAI_CONTROL_PLANE_URL) creates rooms, issues join tickets, and feeds the roster and lifecycle events for a session.
  2. The routing store (SIP_ROUTING_REDIS_URL) is read through ulai-sip-resolver. It is the authority on which project owns a number, which source IPs a trunk accepts, which dispatcher rule matches, and how to reach an outbound trunk. It is also where telecom events are published.
  3. The carrier, reachable at SIP_PUBLIC_IP.

Why a separate service

The SIP stack and the room gateway share no dependencies: the SIP side links no WebRTC, and the room side links no SIP. That keeps the binary — and the image — small, and it is why the only native dependency in the whole thing is libopus, because there is no production-grade pure-Go Opus encoder. Every other piece of the audio path — resampling, μ-law/A-law, RTP, SDP, SRTP — is pure Go.

Where to go next

1.2 - Getting started

Run the image and put a call through it.

The gateway ships as a container image. There is nothing to compile and nothing to install on the host beyond Docker.

What you need

RequirementWhy
A host with a routable public IPIt is advertised to the carrier in SDP and Contact, and media is streamed to it
Docker, with host networking availableThe RTP range cannot be published through a proxy — see networking
A control-plane URLRooms are created and joined there
A routing store URL (Redis)Every call is resolved against it
A SIP trunk configured in the routing storeInbound numbers, outbound termination, or both
A control-plane API keyRequired for POST /sip/originate

Ports that must be reachable from the carrier: 5060 (UDP, TCP or TLS) for signalling and 10000–10500 (UDP) for media. Opening the first without the second gives you calls that connect and then have no audio.

Pull the image

docker pull asia-south1-docker.pkg.dev/arctic-operand-415316/ulai/sip:v6

If the registry is private, authenticate first:

gcloud auth configure-docker asia-south1-docker.pkg.dev

Run it

docker run -d --name ulai-sip \
  -e ULAI_CONTROL_PLANE_URL=https://stgcp.ulai.co.in \
  -e SIP_ROUTING_REDIS_URL='redis://USER:PASSWORD@redis.ulai.co.in:6379' \
  -e SIP_PUBLIC_IP=148.113.58.51 \
  -e SIP_LISTEN_ADDR=0.0.0.0:5060 \
  -e SIP_TRANSPORT=udp \
  -e SIP_RTP_HOST=0.0.0.0 \
  -e SIP_RTP_PORT_LOW=10000 \
  -e SIP_RTP_PORT_HIGH=10500 \
  -e SIP_HTTP_PORT=8082 \
  --network host \
  asia-south1-docker.pkg.dev/arctic-operand-415316/ulai/sip:v6

Three things in that command are load-bearing:

  • --network host is required. Publishing the RTP range with -p spawns hundreds of userland proxies and makes every INVITE arrive from the bridge gateway’s address, which the trunk’s IP ACL then rejects with 403.
  • SIP_PUBLIC_IP must be the host’s own routable address. It is what every SDP tells the carrier to stream media to, which is why 0.0.0.0 is rejected at startup.
  • SIP_RTP_HOST=0.0.0.0 is what the RTP sockets bind to. On most cloud VMs the public IP is attached at a NAT or load-balancer layer and is not on any interface inside the VM, so binding to it fails outright.

Keep secrets out of the host’s process list with an env file instead:

docker run -d --name ulai-sip --network host \
  --env-file /etc/ulai/sip.env \
  asia-south1-docker.pkg.dev/arctic-operand-415316/ulai/sip:v6
# /etc/ulai/sip.env
ULAI_CONTROL_PLANE_URL=https://stgcp.ulai.co.in
SIP_ROUTING_REDIS_URL=redis://USER:PASSWORD@redis.ulai.co.in:6379
SIP_PUBLIC_IP=148.113.58.51
SIP_LISTEN_ADDR=0.0.0.0:5060
SIP_TRANSPORT=udp
SIP_RTP_HOST=0.0.0.0
SIP_RTP_PORT_LOW=10000
SIP_RTP_PORT_HIGH=10500
SIP_HTTP_PORT=8082

Only three variables are genuinely required — ULAI_CONTROL_PLANE_URL, SIP_ROUTING_REDIS_URL and SIP_PUBLIC_IP. Everything else has a default. Every knob is in the configuration reference.

Check it

A healthy start looks like this:

docker logs ulai-sip
[runtime] GOMAXPROCS=4 (cgroup CPU quota=4.00, host cores=8)
[runtime] GOMEMLIMIT=3481MiB (cgroup limit=4096MiB, 85%)
[runtime] GOGC=200
routing store reachable
sip-sfu-gateway: SIP listening on 0.0.0.0:5060/udp public=148.113.58.51 rtp=10000-10500 rtp_timeout=30s
sip-sfu-gateway: HTTP listening on :8082 (control plane https://stgcp.ulai.co.in, ice_servers=1)

If the routing store cannot be reached the gateway still starts, and says so:

WARNING: routing store unreachable (...) — every call will be rejected until it recovers

That is deliberate — a listener that answers with an honest 500 is worth more than one that is not there at all.

Then the two endpoints:

curl -s localhost:8082/health
# {"status":"ok","service":"sip-sfu-gateway"}

curl -s localhost:8082/calls
# {"live":[]}

/health says nothing about the SIP leg. Check that separately — the image ships sipsak for exactly this:

docker exec ulai-sip sipsak -s sip:healthcheck@127.0.0.1:5060

Your first inbound call

  1. Point a number at the gateway in the routing store: the SIP domain the carrier addresses must map to a project, the number must be assigned, the trunk’s IP ACL must include the carrier’s source address, and a dispatcher rule must match.
  2. Dial the number.
  3. Watch the log. Each line of a call is prefixed with the number, so a busy gateway can be read one call at a time:
[+919000000000] [sip-gw] INVITE: to=+919000000000 from=+919111111111 host=sbc.example.com source=13.14.15.16 callID=3e114ad4…
[+919000000000] [sip-gw] routed: project=proj_123 trunk=Main SBC rule=support (direct) agent=support-bot
[+919000000000] [sip-gw] room sess_abc (via resolver, created=true)
[+919000000000] [sip-gw] call accepted — connecting to room sess_abc
[+919000000000] [sfugw] webrtc state: connected

A rejection is equally legible, and the SIP status says which check failed — see troubleshooting.

Your first outbound call

POST /sip/originate needs a control-plane API key and a trunk id. The room is created for you unless you pass session_id:

curl -X POST http://148.113.58.51:8082/sip/originate \
  -H "Authorization: Bearer $ULAI_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "to_number":        "+919363399639",
    "trunk_id":         "74958094-8c76-419f-aa42-d4eac6768ec2",
    "project_id":       "fa3f23f8-a75d-44a4-84f0-6a1c828f579f",
    "from_number":      "+13187184515",
    "name":             "sip-module",
    "max_participants": 8,
    "offer_srtp":       true,
    "require_srtp":     true,
    "dial_verbatim":    false
  }'

Only to_number and trunk_id are required, plus project_id unless ULAI_PROJECT_ID is set. Everything else has a default — including the SRTP policy, which is inferred from the trunk’s transport. Add "session_id": "527f78a3262dbe8b0d0a678a9e98ac29" to dial into a room that already exists instead of creating one.

The response comes back immediately:

{
  "status": "originating",
  "to_number": "+919363399639",
  "session_id": "527f78a3262dbe8b0d0a678a9e98ac29",
  "created": true,
  "name": "sip-module",
  "trunk": "Main SBC"
}

The 202 comes back as soon as the trunk is resolved and the room exists — the phone is still ringing. Point an agent or a browser at session_id while it does. The room is only joined once the callee actually answers, so nothing greets a ringtone.

Stopping it

SIGTERM starts a drain, not a kill: new calls are refused, in-flight calls get up to 60 seconds to finish, and anything left is then cancelled with 5 seconds of grace. Give Docker a matching timeout so it does not SIGKILL through it:

docker stop --time 70 ulai-sip

Where to go next

  • Concepts — what happens between the INVITE and the audio.
  • HTTP API — the full request and response shapes.
  • Operations — networking, observability, troubleshooting.

1.3 - Concepts

How the gateway is put together and how a call moves through it.

Three things are worth understanding before changing anything:

  • Call flows — the ordered steps of an inbound and an outbound call, and why the order differs.
  • Routing plane — how a call is admitted, how a trunk is resolved, and what gets published.
  • Media pipeline — codecs, SRTP, jitter, DTMF, and the ways a call ends when nobody sends a BYE.

1.3.1 - Call flows

The ordered steps of an inbound and an outbound call — and why the orders differ.

Both legs end in the same place: a bridgedCall handed to sfugw.Run, which moves audio until one side stops. Everything before that differs.

Inbound

The gateway is the answering party. telephony.SIPRuntime invokes gateway.OnInvite for every out-of-dialog INVITE.

 carrier            gateway                 resolver        control plane
   │  INVITE ────────▶ │                        │                 │
   │                   │ 1. Resolve ───────────▶│                 │
   │                   │    (host, number, IP)  │                 │
   │  ◀── 4xx/5xx ──── │ ◀── reject ────────────│                 │
   │                   │                        │                 │
   │                   │ 2. CreateSession ──────┼────────────────▶│
   │  ◀── 200 OK ───── │ 3. Accept                                │
   │  ──── ACK ──────▶ │                                          │
   │                   │ 4. PublishEvent CALL_ANSWERED ─▶ resolver│
   │                   │ 5. JoinSession ────────┼────────────────▶│
   │  ◀═══ RTP ══════▶ │ ◀════════ bridge ══════╪═════ WebRTC ═══▶│
  1. Route. (host, dialled number, source IP) → project, trunk ACL, dispatcher rule. A rejection here is a specific SIP status the carrier can act on, not a blanket 503. See routing.
  2. Room. An X-Session-Id header on the INVITE wins if present; otherwise a session is created, carrying the dispatcher rule in its metadata so the platform can see what the room is for. The header only chooses the room — it cannot overrule admission.
  3. Answer. 200 OK + ACK, which binds RTP.
  4. Tell. CALL_ANSWERED is published before the WebRTC handshake, not after. That event is what summons the agent, so the dispatcher gets to work while the gateway is still joining.
  5. Join and bridge. A seat is taken in the room, then audio flows.

The room is joined only after the call is answered. On this path “accepted” is the pickup, so no seat is taken until there is a live call to put in it — a join ticket is only valid for about 30 seconds, and a participant appearing in the room is the agent’s cue to start talking.

Rejections

ConditionSIP status
Empty To user400 Bad Request
Gateway is shutting down503 Shutting down
Unknown domain, unassigned number, missing config404 Not Found
Source IP not in the trunk’s ACL403 Forbidden
No dispatcher rule matched480 Temporarily Unavailable
Routing lookup failed500 Server Internal Error
Session could not be created503 Session unavailable

The project-id lookup is deliberately best-effort: a call that routed cleanly is not dropped because that second lookup missed. The dispatcher loses a field, the caller keeps their call.

Outbound

The gateway is the calling party. POST /sip/originate splits into a synchronous half the caller can act on and an asynchronous half it cannot.

 caller              gateway                 resolver        control plane
   │ POST ──────────▶ │                         │                 │
   │                  │ auth: Bearer <ulai key> │                 │
   │                  │ 1. GetOutboundTrunk ───▶│                 │
   │ ◀── 404/502 ──── │ ◀── not found ──────────│                 │
   │                  │ 2. CreateSession ───────┼────────────────▶│
   │ ◀── 202 ──────── │    (unless session_id given)              │
   │   {session_id}   │                                           │
   ╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌ │ ╌╌ background ╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌ │
   │                  │ 3. INVITE ──▶ trunk, block through ring    │
   │                  │ 4. PublishEvent CALL_ANSWERED ─▶ resolver  │
   │                  │ 5. JoinSession ─────────┼────────────────▶│
   │                  │ ◀════════ bridge ═══════╪═════ WebRTC ═══▶│
  1. Trunk first. An unknown trunk_id must not leave an orphan room behind, so the trunk is resolved before anything is created.
  2. Room. Created unless the request carried a session_id. Its metadata carries the number, the trunk and the agent_id, so an orchestrator watching the control plane’s discovery feed can be ready before the phone is answered.
  3. Dial. Originate blocks through ring, up to OriginateTimeout (60 s). The dial context stays alive for the whole call — sipgo builds the client transaction on it, so cancelling it at answer would pull the dialog out from under the call just connected.
  4. Tell, then 5. join and bridge, exactly as inbound.

Ringing takes up to 30 seconds and has nothing useful to say until it is over, which is why it is not on the request path. Trunk lookup and room creation are fast, can fail in ways the caller can act on, and produce a result the caller needs — so they are.

If the callee never answers

Busy, rejected, no answer, unreachable trunk — the room created for the call is terminated, because anything already waiting in it deserves to be told rather than left listening to silence. A 429 from the trunk is logged specially: it is the one failure that is the platform’s own fault, and it is fixed by dialling slower, not by retrying.

What both legs do at the end

sfugw.Run returns when the SIP leg drops, the room ends, or the context is cancelled. Then:

  • CALL_HANGUP is published, with duration_seconds, pairing the CALL_ANSWERED already sent. It is published on a context detached from the call’s own, because the call’s context is usually being torn down at exactly that moment.
  • The call is removed from GET /calls.
  • The session is terminated, unless the room was borrowed and SIP_TERMINATE_SESSION_ON_HANGUP=false.

1.3.2 - Routing plane

How a call is admitted, how a trunk is resolved, and what gets published.

Everything the gateway knows about numbers, trunks and agents comes from ulai-sip-resolver over the platform’s Redis. The gateway holds no routing table of its own.

inbound  : (host, dialled number, source IP)  →  project, trunk ACL, dispatcher rule
outbound : (project, trunk id)                →  address, transport, digest auth

Neither leg carries trunk details on the wire. Callers used to send host, port, transport and credentials on every origination, which put carrier passwords in every dialler’s logs and meant a trunk migration had to be rolled out to each of them.

Inbound admission

Resolve(host, dialledNumber, sourceIP) is the whole admission decision. It returns the trunk the call came in on and the dispatcher rule that won, or an error that maps to a specific SIP status:

Resolver errorStatusMeans
ErrHostNotFound404No project owns that SIP domain
ErrNumberNotAssigned404The domain is known, the number is not
ErrConfigNotFound404The project has no SIP configuration
ErrIPNotAllowed403The INVITE came from an address the trunk’s ACL does not list
ErrNoMatchingRoute480Admitted, but no dispatcher rule matched the number
anything else500The lookup itself failed

Each one is a different operational problem and deserves to be distinguishable from the carrier’s side.

The project id is a second, separate lookup (ulai:sip_host:<host>), because the resolver’s Result does not carry it. That lookup is best-effort: if it fails the call still proceeds, logged as continuing without a project id, and the published events lose a field.

Session metadata

A room created for an inbound call carries the routing decision in its metadata, which rides the control plane’s discovery feed. An orchestrator that has never heard of the call can read it and know which agent the room is waiting for:

KeySource
project_idThe host lookup
trunk_id, trunk_nameThe resolved trunk
rule_id, rule_name, rule_typeThe dispatcher rule
agent_nameThe rule, when it names one
dispatch_metadataThe rule’s free-form metadata, JSON-encoded
name, phone_number, dialled, sip_call_idThe call
directioninbound
sourcesip-sfu-gateway

Empty fields are left out rather than written blank. Several of them routinely are — rules are stored keyed by id with no id inside the value — and a dispatcher can act on an absent rule_id where it cannot tell a blank one from a real empty answer.

An outbound room’s metadata is the same idea with the outbound fields: phone_number, caller_id, name, agent_id, project_id, trunk_id, trunk_name, direction: outbound, source.

Outbound trunks

GetOutboundTrunk(projectID, trunkID) returns a stored trunk, which the gateway reduces to dial parameters:

  • Address — accepted as host, host:port, or either with a sip:/sips: scheme, userinfo or URI parameters attached. Port 0 means unspecified, and the dialler fills in 5061 for TLS or 5060 otherwise.
  • Transport — udp, tcp, tls, or empty (treated as UDP). Anything else is an error rather than a silent fallback: dialling a TLS-only trunk over UDP fails as a timeout minutes later, which is a miserable way to learn about a typo.
  • Credentials — digest username and password, empty when the trunk authenticates by IP ACL.

Media encryption is inferred from transport

The stored trunk carries a transport but no media-encryption field, so SRTP policy is derived:

Trunk transportOffer SRTPRequire SRTP
tlsyesno
udp, tcpnono

This is not cosmetic. A carrier with secure trunking enabled — Twilio’s is, on a TLS trunk — answers an RTP/AVP offer with 488 Secure media required and the call never rings.

Offering without requiring is the safe half: a carrier that wants SRTP finds crypto in the offer; one that does not echoes no a=crypto and the call falls back to plain RTP. Override per request with offer_srtp and require_srtp — including offer_srtp: false to force cleartext media on a TLS trunk. require_srtp implies offer_srtp; asking for the contradictory pair (offer_srtp: false, require_srtp: true) is a 400.

Events

Two event types are published to the platform’s telecom stream, through the same resolver that authorised the call:

TypeWhenNotable metadata
CALL_ANSWEREDImmediately after the call is answered, before the room is joinedrouting fields, session_id, sip_call_id, direction, to_number, from_number
CALL_HANGUPWhen the bridge endsthe same, plus duration_seconds

Publishing is bounded at 2 seconds and best-effort: an event is never allowed to hold up a call, and a miss is logged and nothing more. The hangup publish is detached from the call’s context on purpose — that context is being cancelled at precisely the moment the event matters most.

Startup probe

go-redis connects lazily, so without a probe the first sign of a dead routing store would be a carrier receiving a 500. At startup the gateway looks up a host no project can own and reports what happened:

routing store reachable
WARNING: routing store unreachable (...) — every call will be rejected until it recovers

It does not fail startup. A malformed URL already did that; a store that is down now may be up a second from now.

1.3.3 - Media pipeline

Codecs, SRTP, jitter, DTMF, and the ways a call ends without a BYE.

The audio path

The carrier speaks 8 kHz G.711. The SFU speaks 48 kHz Opus. Everything between the two happens inside the gateway:

uplink   caller μ-law 8k ─decode─▶ PCM 8k ─resample─▶ PCM 48k ─encode─▶ Opus ─▶ room
downlink room Opus 48k ─decode─▶ PCM 48k ─resample─▶ PCM 8k ─mix─▶ μ-law ─▶ caller

Both directions run on a 20 ms cadence: 160 μ-law bytes on the wire, 960 samples per Opus frame. Opus is encoded at 24 kbit/s with complexity 5 — wideband speech without maxing out a CPU that may be carrying many concurrent calls.

The downlink mixes: a room can hold several publishers, and the caller gets all of them summed into one stream. Resampling is go-audio-resampler (pure Go, SIMD-enabled); μ-law and A-law conversion is zaf/g711. The only native code in the binary is libopus.

Codec negotiation

The SDP profile is deliberately narrow — the audio pipeline is μ-law 8 kHz, so accepting anything else would be a lie:

  • PCMU (μ-law, PT 0) or PCMA (A-law, PT 8). When the carrier negotiates A-law, the RTP loops transcode A-law↔μ-law at the wire boundary so everything above stays μ-law.
  • telephone-event (RFC 4733 DTMF) on whatever payload type the carrier assigns.
  • No Opus on the SIP side, no video, no multiple m= lines.

SRTP

SDES only, with the two common profiles: AES_CM_128_HMAC_SHA1_80 (Twilio’s default) and _32. Whether it is offered is decided by the trunk’s transport — see routing.

Inbound RTP: reorder, dedupe, conceal

A conventional jitter buffer imposes a fixed delay on every packet. For a voice agent that delay is charged to time-to-first-token on every turn, including the overwhelming majority where the network was perfectly ordered. So the inbound buffer is not a fixed-delay design:

Packet arrivesWhat happens
In sequenceEmitted immediately, zero added latency
Out of orderHeld only until the gap resolves, bounded by a 40 ms holdout
DuplicatedDropped
NeverConcealed after the holdout, stream continues

Concealment repeats the previous frame, attenuated, decaying to silence over a few frames. Repeating preserves the spectral envelope so an ASR hears a brief smear rather than the click-and-jump digital silence produces; decaying stops a lost burst becoming an audible buzz.

The cost is paid only by calls that actually have a disordered network.

Outbound RTP: a small playout cushion

The provider’s writer and the RTP writer are two independent 20 ms tickers. Without a cushion, ordinary scheduler jitter forces silence into the middle of speech. The playout buffer builds 60 ms (3 frames) before starting and caps added latency at 160 ms (8 frames), dropping the oldest beyond that.

Symmetric RTP and the source gate

Carriers behind NAT routinely send RTP from a port they never advertised in SDP, so the first well-formed packet latches the peer address and the writer re-targets to it. Everything after that is checked against the latch.

That check matters because the RTP port range is a few hundred even ports cycled round-robin: a call that ends while its carrier is still streaming leaves packets in flight that land on whichever call binds that port next. Before the gate existed, they were decoded and mixed into a live conversation as a second voice.

A genuine media re-anchor — a B2BUA leg swap, an SBC failover — is admitted only after the new source proves persistence (5 packets over at least 200 ms). A re-INVITE can pre-authorise an address, but the previous peer stays valid until the new one actually speaks: a re-INVITE is an intention, and a peer that never follows through must not be able to mute a working call.

Re-INVITE, hold, and session timers

sipgo’s OnInvite fires for every INVITE, including in-dialog ones. Handing those to the application handler treats a mid-call re-INVITE — a session-timer refresh, a hold, a media re-anchor, an SBC failover — as a brand new call: a second agent, a second billing row, a 180 Ringing inside an established dialog, a 200 OK with a new To tag (a protocol violation), and an answer advertising a new RTP port, so the carrier moves media to a socket nobody reads. Dead air for the rest of the call.

In-dialog INVITEs are therefore handled separately, and answered as a re-statement rather than a negotiation: same To tag, same RTP port, same codec, session id unchanged with its version bumped, and the direction attribute mirrored so hold is acknowledged rather than contradicted. The only thing that may legitimately change is where the peer wants media — and that is followed.

A re-INVITE that tries to switch G.711 flavour mid-call is refused rather than silently answered with a lie: the provider’s transcode setting is fixed when it is constructed.

Session timers (timer) are supported; an INVITE that Requires an extension the gateway does not support is rejected rather than answered.

DTMF

RFC 4733 telephone-event packets are decoded and surfaced as events. The gateway logs them (DTMF: 5) and does nothing else with them — it has no IVR of its own. An agent in the room that wants digits should consume the session feed, not expect the gateway to act.

Ending a call without a BYE

A hangup’s BYE can be lost. A TLS trunk calling back a UDP-only listener never reaches the gateway at all, and without a backstop such a call — and its room — stays up until the process exits.

SIP_RTP_TIMEOUT_SECONDS (default 30) ends a call whose inbound audio has stopped for that long, as if the far end had hung up. It is paused while the call is on hold, so a legitimately silent leg is not cut off. Set 0 to disable.

The other end-of-call signals:

SignalSource
BYEThe far end, normally
session_terminatedThe control plane, over the session events feed — the authoritative “this call is really over”
Transport Done()The WebRTC leg dropping, which may be a transient blip the SDK reconnects through
Drain cancellationShutdown, after the 60 s drain window

1.4 - Configuration

Every environment variable, its default, and what breaks when it is wrong.

Configuration is read from the environment once, at startup, into a single validated struct. Everything downstream reads that struct, never the environment — which is also why a test can construct one directly.

Validation reports every problem at once, so a misconfigured deployment does not have to be fixed one restart at a time.

Required

VariableDescription
ULAI_CONTROL_PLANE_URLBase URL of the Ulai control plane, e.g. https://stgcp.ulai.co.in. Rooms are created, joined and terminated here.
SIP_ROUTING_REDIS_URLThe routing store every call is resolved against. REDIS_URL is accepted as a fallback.
SIP_PUBLIC_IPThe address advertised in SDP and Contact. Must be routable by the carrier. 0.0.0.0 is rejected — it is not a listen address, it is what the carrier is told to stream media to.

SIP signalling

VariableDefaultDescription
SIP_LISTEN_ADDR0.0.0.0:5060What sipgo binds to. host:port; the host may be 0.0.0.0.
SIP_TRANSPORTudpudp, tcp or tls.
SIP_TLS_CERT_PATH—Required when SIP_TRANSPORT=tls.
SIP_TLS_KEY_PATH—Required when SIP_TRANSPORT=tls.
SIP_SESSION_HEADERX-Session-IdThe INVITE header that may point a call at an existing room. Must not be blank.

RTP media

VariableDefaultDescription
SIP_RTP_HOST0.0.0.0What RTP sockets bind to, as distinct from SIP_PUBLIC_IP, which is what is advertised. On most cloud VMs the public IP is attached at a NAT or load-balancer layer and is not on any interface inside the VM, so binding to it fails with cannot assign requested address. The default is right nearly everywhere.
SIP_RTP_PORT_LOW10000Bottom of the RTP allocation range.
SIP_RTP_PORT_HIGH10500Top of the range. Must be above the low end, and both within 1–65535. Even ports only, by RTCP convention.
SIP_RTP_TIMEOUT_SECONDS30End a call whose inbound audio has stopped for this long, as if the far end had hung up — the backstop for a BYE that never arrives. Paused while the call is on hold. 0 disables it.

HTTP API

VariableDefaultDescription
SIP_HTTP_PORT8082Port for /health, /calls and /sip/originate. Chosen clear of the worker (8080) and the Tata gateway (8081).

Rooms

VariableDefaultDescription
ULAI_PROJECT_ID—Default project for POST /sip/originate when the body omits project_id. The single-tenant deployment’s answer to sending it every time.
ULAI_MAX_PARTICIPANTS8Size of a room the gateway creates. Overridden per request by max_participants. Must be positive.
SIP_TERMINATE_SESSION_ON_HANGUPtrueEnd the whole session — every participant, every transport — when the SIP leg drops, rather than merely leaving the room. Set false only when a room reached via X-Session-Id or session_id is genuinely shared. A room the gateway created is always terminated regardless.

ICE

The gateway is STUN-only by default (stun:stun.l.google.com:19302), which is fine on a public host and will fail behind symmetric NAT.

VariableDefaultDescription
TURN_URLS—Comma-separated TURN URLs. Appended to the STUN default.
TURN_USERNAME—TURN credential.
TURN_PASSWORD—TURN credential.

Go runtime

At startup the process aligns the Go runtime with its cgroup limits — two defaults that are correct on a bare host become latency or OOM bugs inside a container. An explicit environment variable always wins.

VariableBehaviour when unset
GOMAXPROCSPinned to ceil(cgroup CPU quota) when that is below the host core count. Left alone otherwise.
GOMEMLIMITSet to 85% of the cgroup memory limit, or of total system RAM when there is none. The headroom covers goroutine stacks and the off-heap libopus and resampler allocations GOMEMLIMIT cannot see.
GOGCRaised to 200, trading memory for fewer GC cycles and less latency jitter, bounded by the limit above.

Each decision is logged at startup:

[runtime] GOMAXPROCS=4 (cgroup CPU quota=4.00, host cores=8)
[runtime] GOMEMLIMIT=3481MiB (cgroup limit=4096MiB, 85%)
[runtime] GOGC=200

Built-in timeouts

Not configurable — they are constants, listed here because they explain the behaviour you will observe.

ConstantValueBounds
ResolveTimeout5 sOne routing-store lookup
CreateTimeout15 sCreating a room
JoinTimeout15 sJoining a room
DialTimeout15 sThe WebRTC signalling dial
OriginateTimeout60 sAn outbound INVITE, through ring
TerminateTimeout5 sTerminating a room, or hanging up a leg
DrainTimeout60 sThe shutdown drain
CancelGrace5 sExtra time after the drain deadline cancels calls
MaxRequestBytes8 KiBAn originate body — it is a control API, not an upload
publish timeout2 sOne event publish, best-effort
control-plane HTTP timeout10 sAny single control-plane request

Example

# Required
ULAI_CONTROL_PLANE_URL=https://stgcp.ulai.co.in
SIP_ROUTING_REDIS_URL=redis://user:password@redis.example.com:6379/0
SIP_PUBLIC_IP=203.0.113.10

# SIP + RTP
SIP_LISTEN_ADDR=0.0.0.0:5060
SIP_TRANSPORT=udp
SIP_RTP_PORT_LOW=10000
SIP_RTP_PORT_HIGH=10500
SIP_RTP_TIMEOUT_SECONDS=30

# HTTP
SIP_HTTP_PORT=8082

# Rooms
SIP_TERMINATE_SESSION_ON_HANGUP=true

1.5 - HTTP API

/health, /calls and /sip/originate.

Three endpoints, served on SIP_HTTP_PORT (default 8082). Only /sip/originate is authenticated.

MethodPathAuth
GET/healthnone
GET/callsnone
POST/sip/originateAuthorization: Bearer <ulai api key>

GET /health

curl -s http://148.113.58.51:8082/health
{ "status": "ok", "service": "sip-sfu-gateway" }

Always 200 while the process is serving. It says nothing about the SIP leg or the routing store — see observability for a check that covers both.

GET /calls

Every call currently bridged, oldest first.

curl -s http://148.113.58.51:8082/calls
{
  "live": [
    {
      "to_number": "+919363399639",
      "from_number": "+13187184515",
      "name": "sip-module",
      "sip_call_id": "3e114ad4-a9ad-4a0b-97b9-5cf11190bdd8",
      "session_id": "527f78a3262dbe8b0d0a678a9e98ac29",
      "project_id": "fa3f23f8-a75d-44a4-84f0-6a1c828f579f",
      "direction": "outbound",
      "source": "originate",
      "started_at": "2026-09-21T10:14:52.118Z"
    }
  ]
}
FieldMeaning
to_number, from_numberNormalised E.164
nameDisplay name in the room roster
sip_call_idThe SIP Call-ID — the join key between logs, events and carrier CDRs
session_idThe Ulai room
project_idOwning project; may be empty inbound if the host lookup missed
directioninbound or outbound
sourceoriginate, resolver, or the session header’s name when a room was supplied on the INVITE
started_atWhen the bridge started, RFC 3339

The list is sorted rather than map-ordered, so polling it does not reshuffle rows on every request.

POST /sip/originate

Places one outbound call. Authenticates with a control-plane API key, and the room opened for the call belongs to the project that key identifies — which is how one gateway serves several projects without per-project configuration.

curl -X POST http://148.113.58.51:8082/sip/originate \
  -H 'Authorization: Bearer ulai_live_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' \
  -H 'Content-Type: application/json' \
  -d '{
    "to_number":        "+919363399639",
    "trunk_id":         "74958094-8c76-419f-aa42-d4eac6768ec2",
    "project_id":       "fa3f23f8-a75d-44a4-84f0-6a1c828f579f",
    "from_number":      "+13187184515",
    "name":             "sip-module",
    "max_participants": 8,
    "offer_srtp":       true,
    "require_srtp":     true,
    "dial_verbatim":    false,
    "session_id":       "527f78a3262dbe8b0d0a678a9e98ac29"
  }'

The smallest request that works — the gateway creates the room and infers SRTP policy from the trunk’s transport:

curl -X POST http://148.113.58.51:8082/sip/originate \
  -H "Authorization: Bearer $ULAI_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "to_number": "+919363399639",
    "trunk_id":  "74958094-8c76-419f-aa42-d4eac6768ec2",
    "project_id":"fa3f23f8-a75d-44a4-84f0-6a1c828f579f"
  }'

Request body

FieldTypeRequiredDescription
to_numberstringyesDestination, E.164. Normalised before dialling.
trunk_idstringyesOutbound trunk within the project. The resolver turns it into an address, a transport and credentials — none of which are ever sent on the wire.
project_idstringunless ULAI_PROJECT_ID is setScopes trunk_id.
session_idstringnoDial into a room that already exists. Omit it and the gateway creates one and returns its id.
from_numberstringnoCaller ID presented to the trunk.
namestringnoDisplay name in the room roster. Defaults to to_number.
agent_idstringnoRecorded in session metadata and visible on the control plane’s discovery feed, so an orchestrator can tell which agent the room is waiting for. Unused by the gateway otherwise.
max_participantsintnoSizes a newly created room. Ignored when session_id is given. Defaults to ULAI_MAX_PARTICIPANTS.
offer_srtpboolnoOffer RTP/SAVP with an a=crypto line. Defaults to true on a TLS trunk, false otherwise.
require_srtpboolnoFail the call if the answer carries no crypto. Implies offer_srtp.
dial_verbatimboolnoSend to_number without a leading +. For self-hosted or CUSTOM trunks whose dialplan matches literal digit patterns; hosted ITSPs want E.164-with-plus.

Bodies are capped at 8 KiB. Validation reports every problem at once, so a dialler does not get one complaint per round trip.

202 Accepted

{
  "status": "originating",
  "to_number": "+919363399639",
  "session_id": "527f78a3262dbe8b0d0a678a9e98ac29",
  "created": false,
  "name": "sip-module",
  "trunk": "Main SBC"
}

created says whether the gateway made the room (and is therefore responsible for tearing it down). The response returns as soon as the trunk is resolved and the room exists — the phone is still ringing. Point an agent or a browser at session_id while it does; the gateway itself joins the room only once the callee answers.

Errors

StatusCause
400Malformed JSON, failed validation, or no project_id and no ULAI_PROJECT_ID
401Missing or malformed Authorization: Bearer
404No such trunk for that project, or no SIP config for the project
502Room creation failed, or the stored trunk address is malformed
503The gateway is shutting down
{ "error": "to_number is required\ntrunk_id is required" }

A 404 means the request asked for something that is not there; a 502 means an upstream failed. Only one of those is worth retrying.

What happens next

Nothing else comes back over HTTP. The call’s progress shows up in three places:

  • The log, prefixed with the number.
  • GET /calls, once the bridge starts.
  • Telecom events — CALL_ANSWERED then CALL_HANGUP, published to the platform’s stream. See routing.

If the callee never answers, the room created for the call is terminated rather than left running.

1.6 - Operations

Deploying it, networking it, watching it, and fixing it.
  • Deployment — container, compose, and the CI pipeline.
  • Networking — ports, host networking, and why port publishing breaks calls.
  • Observability — logs, health checks, and the control-plane wire log.
  • Troubleshooting — symptom first, cause second.

1.6.1 - Deployment

Running the image on a host.

The gateway is distributed as a container image. A deployment is one container on a host with a routable public IP, run with host networking.

docker run

docker run -d --name ulai-sip \
  -e ULAI_CONTROL_PLANE_URL=https://stgcp.ulai.co.in \
  -e SIP_ROUTING_REDIS_URL='redis://USER:PASSWORD@redis.ulai.co.in:6379' \
  -e SIP_PUBLIC_IP=148.113.58.51 \
  -e SIP_LISTEN_ADDR=0.0.0.0:5060 \
  -e SIP_TRANSPORT=udp \
  -e SIP_RTP_HOST=0.0.0.0 \
  -e SIP_RTP_PORT_LOW=10000 \
  -e SIP_RTP_PORT_HIGH=10500 \
  -e SIP_HTTP_PORT=8082 \
  --network host \
  asia-south1-docker.pkg.dev/arctic-operand-415316/ulai/sip:v6

Three things in that command are load-bearing:

  • --network host — see networking. Publishing the RTP range with -p spawns hundreds of userland proxies and rewrites the INVITE’s source address, which the trunk ACL then rejects with 403.
  • SIP_PUBLIC_IP — the address the carrier is told to send media to. It must be the host’s own routable address, and never 0.0.0.0.
  • SIP_RTP_HOST=0.0.0.0 — what the RTP sockets bind to. On a cloud VM the public IP usually is not on any interface, so binding to it fails outright.

Pass secrets with --env-file rather than -e if the host’s process list is readable:

docker run -d --name ulai-sip --network host \
  --env-file /etc/ulai/sip.env \
  asia-south1-docker.pkg.dev/arctic-operand-415316/ulai/sip:v6

docker compose

services:
  sip-gateway:
    image: asia-south1-docker.pkg.dev/arctic-operand-415316/ulai/sip:v6
    container_name: ulai-sip
    network_mode: host
    env_file:
      - .env
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "./health-check.sh"]
      interval: 30s
      timeout: 5s
      start_period: 15s
      retries: 3

network_mode: host replaces a ports: block — the container binds the host’s ports directly, which is the only arrangement that keeps the carrier’s source address intact and the RTP range usable.

docker compose up -d
docker compose logs -f

What the image contains

A Debian slim runtime with the gateway binary, libopus, ca-certificates, and curl plus sipsak for the health check. It runs as an unprivileged user (sipgw, uid 10001) — SIP 5060 and the RTP range are unprivileged ports, so nothing here needs root.

Exposed ports:

PortProtocolPurpose
8082TCPHTTP API
5060UDP, TCPSIP signalling
10000–10500UDPRTP media

Upgrading

Pull the new tag, then replace the container. There is no state on disk — every call is in memory and every routing decision is read fresh — so a replacement is a restart, not a migration:

docker pull asia-south1-docker.pkg.dev/arctic-operand-415316/ulai/sip:v7
docker stop --time 70 ulai-sip
docker rm ulai-sip
docker run -d --name ulai-sip --network host --env-file /etc/ulai/sip.env \
  asia-south1-docker.pkg.dev/arctic-operand-415316/ulai/sip:v7

Live calls do not survive the replacement beyond the drain window below, so prefer a quiet period — or run a second host and move the trunk across.

Restart and shutdown

SIGINT/SIGTERM starts a drain rather than a kill:

shutdown signal received — draining for up to 1m0s

New calls are refused (503 on both legs), in-flight calls get up to 60 seconds to finish, and anything still up is then cancelled with a further 5 seconds of grace. The routing-store connection is closed last, so a draining call’s CALL_HANGUP still has somewhere to go.

Give the container a matching stop timeout so Docker does not SIGKILL through the drain:

docker stop --time 70 ulai-sip

1.6.2 - Networking

Ports, host networking, and why port publishing breaks calls.

Ports

PortProtocolPurpose
8082TCPHTTP API — /health, /calls, /sip/originate
5060UDP and/or TCPSIP signalling (5061 conventionally for TLS)
10000–10500UDPRTP media, one even port per call

The RTP range sizes the gateway’s concurrency: roughly 250 simultaneous calls at the default range, since allocation uses even ports only by RTCP convention. Widen SIP_RTP_PORT_LOW/HIGH to carry more.

The carrier must be able to reach SIP_PUBLIC_IP on the SIP port and the whole RTP range. A firewall that allows 5060 but not 10000–10500 produces a call that connects and then has no audio — the single most common misconfiguration.

Port publishing versus host networking

Run the container with --network host.

Publishing the ports with -p breaks the gateway in two separate ways:

  1. 501 userland proxies. -p 10000-10500:10000-10500/udp spawns one docker-proxy process per port. That is slow to start, heavy at rest, and adds a hop to every media packet.
  2. The trunk ACL rejects the call. A proxied INVITE arrives from the bridge gateway’s address, not the carrier’s. The resolver checks the source IP against the trunk’s ACL, does not find it, and answers 403 Forbidden.

The same reasoning is why SIP_PUBLIC_IP must be a routable address: it is copied into every SDP and Contact the gateway sends, and it is where the carrier will stream media.

Bind address versus advertised address

Two different variables, routinely confused:

VariableMeaning
SIP_RTP_HOST (default 0.0.0.0)What RTP sockets bind to, inside the machine
SIP_PUBLIC_IP (required)What the gateway advertises to the carrier

On AWS, GCP, Azure and most VPS providers the public IP is attached at a NAT or load-balancer layer and is not present on any interface inside the VM. Binding to it fails with cannot assign requested address, so the default 0.0.0.0 is the right answer nearly everywhere.

NAT and ICE

The room side is WebRTC, so it needs ICE. The gateway is STUN-only by default (stun:stun.l.google.com:19302), which works on a public host and fails behind symmetric NAT. Add TURN there:

-e TURN_URLS='turn:turn.example.com:3478?transport=udp,turns:turn.example.com:5349' \
-e TURN_USERNAME=ulai \
-e TURN_PASSWORD=...

The startup line reports how many ICE servers were configured:

sip-sfu-gateway: HTTP listening on :8082 (control plane https://stgcp.ulai.co.in, ice_servers=1)

ice_servers=1 means STUN only.

Symmetric RTP

Carriers behind NAT routinely send RTP from a port they never advertised in SDP. The gateway latches onto the source of the first well-formed packet and sends there, rather than trusting the SDP address — without which roughly half of inbound calls would be one-way. See media.

Transport choice

SIP_TRANSPORTNotes
udpThe default, and what most trunks use
tcpUseful where large INVITEs fragment
tlsRequires SIP_TLS_CERT_PATH and SIP_TLS_KEY_PATH; implies offering SRTP on outbound trunks whose transport is also TLS

A TLS trunk calling back a UDP-only listener is a real failure mode: the BYE never arrives and the call would hang forever, which is what SIP_RTP_TIMEOUT_SECONDS exists to catch.

1.6.3 - Observability

Logs, health checks, and the control-plane wire log.

Reading the log

Every line of a call is prefixed with the number and the leg it arrived on, so a busy gateway’s log can be read one call at a time:

[+919363399639] [sip-originate] call answered — call_id=3e114ad4…, joining room 527f78a3…
[+919363399639] [sfugw] webrtc state: connected
[+919363399639] [sfugw] DTMF: 5
[+919363399639] [sip-originate] call finished (session 527f78a3…, call_id 3e114ad4…)
PrefixEmitted by
[sip-gw]The inbound path
[sip-originate]The outbound path
[sfugw]The room/WebRTC bridge
[control-plane]The control-plane wire log
[runtime]Startup runtime tuning

grep on the number gives one call; grep on the sip_call_id joins the gateway’s log to carrier CDRs and to the published telecom events.

Health checks

There are two legs, and /health only covers one:

curl -fsS http://127.0.0.1:8082/health
sipsak -s sip:healthcheck@127.0.0.1:5060

health-check.sh in the repo runs both and exits non-zero if either fails. It is what docker-compose.yml and the CI verification step use. Only one HEALTHCHECK can be active per image, which is why both checks live in one script.

/health returns 200 whenever the process is serving. It does not assert that the routing store is reachable — that is reported at startup and again on every call that fails because of it.

Live calls

watch -n2 'curl -s localhost:8082/calls | jq'

The list is sorted oldest-first, so rows do not reshuffle between polls. It is also the number the drain reports:

drain deadline reached with 3 call(s) still up — cancelling them

The control-plane wire log

Every request the gateway makes to the control plane — create, join, terminate — is logged with its headers and body:

[control-plane] → POST https://stgcp.ulai.co.in/api/v1/sessions headers={Authorization: Bearer ulai…kzM (51 chars), Content-Type: application/json, X-System-Secret: ahd8…3dg (26 chars)} body={"max_participants":8,…}
[control-plane] ← POST /api/v1/sessions 201 Created (128ms)

It exists because “what did we actually send?” is a question best answered by the log, not by reading the SDK — which matters most when the answer is a 403 and the question is which header was missing.

Three rules it follows:

  • Credentials are masked, never printed whole: first and last four characters plus the length. Enough to tell which secret went out without putting it in every log shipper that reads this process’s output.
  • Auth headers are always reported, present or not. For an auth failure, Authorization: <not sent> is the single most useful thing a log can say.
  • Only failure bodies are logged. A success body carries the join ticket and events token, which are credentials in their own right; a failure body is the control plane’s explanation.

Events

CALL_ANSWERED and CALL_HANGUP are published to the platform’s telecom stream for every answered call, carrying the routing decision, the room, the numbers and — on hangup — duration_seconds. They are the record to build dashboards and alerting on; the gateway keeps no history of its own beyond the live table. See routing.

Startup lines worth alerting on

routing store reachable
WARNING: routing store unreachable (...) — every call will be rejected until it recovers

The gateway deliberately starts either way: a store that is down now may be up a second from now, and a SIP listener answering an honest 500 is worth more than one that is not there at all.

1.6.4 - Troubleshooting

Symptom first, cause second.

The gateway will not start

sip-sfu-gateway: ULAI_CONTROL_PLANE_URL is required (e.g. https://stgcp.ulai.co.in)
SIP_PUBLIC_IP must be a routable address, not 0.0.0.0 — it is advertised to the carrier

Configuration is validated once, and every problem is reported together — fix them all, then restart. 0.0.0.0 in SIP_PUBLIC_IP is rejected on purpose: it is not a listen address, it is what the carrier is told to stream media to.

Other startup failures:

MessageCause
SIP_TLS_CERT_PATH and SIP_TLS_KEY_PATH are required when SIP_TRANSPORT=tlsTLS transport with no certificate
SIP_RTP_PORT_LOW/HIGH (…) must be a valid ascending port rangeInverted or out-of-range ports
routing service: …The Redis URL is malformed — this one does fail startup
SIP runtime init: …The SIP port is already bound, or the TLS material is unreadable

Calls are rejected

The SIP status tells you which check failed:

StatusMeaningWhere to look
403 ForbiddenSource IP not in the trunk’s ACLThe carrier’s signalling address, and whether you are behind a Docker port proxy — see networking
404 Not FoundUnknown SIP domain, unassigned number, or no project configThe routing store: is the domain mapped, is the number assigned
480 Temporarily UnavailableRouted, but no dispatcher rule matchedThe project’s dispatcher rules
500 Server Internal ErrorThe routing lookup itself failedRedis connectivity — WARNING: routing store unreachable will be in the log
503 Session unavailableThe control plane would not create a roomThe [control-plane] wire log
503 Shutting downDrain in progressDeploy timing

Every rejection is logged with the reason immediately before it:

[+919363399639] [sip-gw] rejecting 403: source IP not allowed by the trunk ACL: …

The call connects but there is no audio

Almost always the media path, not the signalling path:

  1. The RTP range is not open. 5060 allowed, 10000–10500 blocked, is the classic. Check the firewall and the security group.
  2. SIP_PUBLIC_IP is wrong. The carrier is streaming to whatever the SDP said. Check it against the host’s actual routable address.
  3. Port publishing instead of host networking. Media arrives at the proxy rather than the process.
  4. One-way only? That is usually NAT on the carrier’s side. The gateway already latches onto the observed source address; if it is still one-way, the outbound direction is being dropped upstream.

Outbound calls fail

ResponseCause
401No Authorization: Bearer header, or a malformed one
400 project_id is required (or set ULAI_PROJECT_ID)Neither the body nor the environment names a project
404That trunk_id does not exist in that project
502Room creation failed, or the stored trunk address is malformed

If the 202 comes back but the phone never rings, the failure is in the background half and only the log has it:

[+919363399639] [sip-originate] Originate failed: …
[+919363399639] [sip-originate] RATE LIMITED by trunk … (SIP 429, Retry-After 30s) — this gateway does not pace originations; the caller must slow down

The room created for that call is terminated rather than left running.

488 Secure media required

A carrier with secure trunking enabled answered a plain RTP/AVP offer. The gateway infers SRTP from the trunk’s transport — TLS trunks offer it, UDP and TCP do not — so either fix the stored transport, or force it per request with "offer_srtp": true. See routing.

404 from the trunk on a self-hosted SBC

A dialplan matching literal digit patterns will not match a leading +. Send "dial_verbatim": true. Hosted ITSPs want the opposite — E.164 with the plus.

A call never ends

If the far end’s BYE is lost — a TLS trunk calling back a UDP-only listener never reaches us at all — the call and its room would stay up until the process exits. SIP_RTP_TIMEOUT_SECONDS (default 30) is the backstop: a call whose inbound audio stops for that long is ended as if the far end had hung up. It is paused while the call is on hold. Setting it to 0 disables the backstop entirely, which is rarely what you want.

The agent talks to a ringtone

It should not — the gateway joins the room only after the callee answers, precisely so a participant appearing in the roster is a reliable cue. If it happens, something else joined the room early; the gateway’s own join is logged as call answered — … joining room ….

The agent keeps talking after the caller hangs up

The session is not being terminated with the call. A room the gateway created is always torn down; a room it was pointed at (X-Session-Id, or session_id on originate) is torn down too unless SIP_TERMINATE_SESSION_ON_HANGUP=false. Check that variable first.

Duplicate agents on one call

A mid-call re-INVITE handled as a new call produces exactly this — a second agent, a second model session, a second billing row. In-dialog INVITEs are handled separately for that reason. If you see it, capture the SIP flow and check whether the second invocation followed a re-INVITE from the carrier’s SBC.

2 - Agent Server

The voice engine, shipped as a container image.

ulai-agent-server is the voice engine. It joins an Ulai SFU room, connects an AI backend, and runs the conversation — the audio chain, turn detection, barge-in, the silence ladder and the end-call contract.

It ships as a container image. There is nothing to build and nothing to compile.

It holds no database, no agent store and no credentials. Everything about how a call runs — the prompt, the voice, the GCP project, the service-account key — arrives from whatever drives it, per call. That is why the configuration below is so short.

Run it

docker run -d --name agent_server --restart unless-stopped \
  -p 127.0.0.1:50052:50052 \
  -p 8000:8000 \
  -e APP_GRPC_LISTEN_PORT=50052 \
  -e APP_HTTP_PORT=8000 \
  -e ULAI_GRPC_API_KEY='<64-character key>' \
  asia-south1-docker.pkg.dev/arctic-operand-415316/ulai/agent_server:v1.02

Check it:

curl -s http://localhost:8000/health     # → ok

That is the whole installation. See deploy it for the full walk-through, including the part that actually makes calls happen.

What you still need

The agent server does not place calls by itself. It waits on port 50052 for an agent client to tell it which room to join and which agent to run.

  agent client  ──gRPC :50052──▶  agent server  ──▶  Ulai SFU room
  (drives calls)                   (this image)       (the conversation)

Both sides must share the same key: whatever you set as ULAI_GRPC_API_KEY here, the client must present. Nothing happens until a client connects.

In this section

  • Overview — what it does, what it deliberately does not, and where it sits.
  • Deploy it — pull, run, verify, and connect a client.
  • Configuration — every environment variable, and the longer list of things that are not environment.
  • What a call carries — why the environment is short, and what arrives per call instead.
  • Interfaces — the two ports and what talks to them.
  • Operations — deployment, observability and troubleshooting.

2.1 - Overview

What the engine does, what it refuses to do, and where it sits.

What it is

The Ulai Agent Server is a single Go binary (orchestrator) that runs one side of a voice conversation. It joins an Ulai SFU room as an ordinary participant, connects an AI backend to that room, and manages the conversation until somebody ends it.

Everything it owns is timing-critical:

  • The audio chain — pre-clean, denoise, near-field foreground isolation, neural VAD.
  • Turn detection — who has the floor, and when that changes.
  • Barge-in — cutting the agent mid-sentence when the caller speaks, and accounting for how much of the turn was actually heard.
  • The silence ladder — nudges when a caller goes quiet, and a hangup when they stay quiet.
  • The end-call contract — the evidence a model must produce before it is allowed to hang up on a human.

What it is not

It is not a place where anything is stored. There is no database, no agent table, no recording bucket and no credential file. A call’s configuration arrives in the request that starts it, and everything worth keeping leaves on the event stream for somebody else to write down.

That is a deliberate constraint rather than an omission. It is what lets one engine serve agents belonging to different customers, billed to different GCP projects, running on different AI vendors — and it is what makes the binary safe to hand to somebody else.

It ownsSomebody else owns
Audio, turns, barge-in, playback timingAgent configuration and where it is stored
The AI session and its lifecycleWhich GCP project and key that session uses
Tool dispatch and the end-call contractWhat a custom tool actually does
Emitting call eventsWriting call records, costs and transcripts

Where it sits

  agent client  ──gRPC──▶  agent server  ──WebSocket──▶  Ulai SFU room
  (your logic)              (this)                        (the conversation)
       │                       │
       │                       └──▶ AI backend (Gemini Live, …)
       │
       └──▶ your database, your dashboards, your billing

The agent client decides which agent runs, what it says, and where it is billed. It sends all of that with each call. The agent server runs the call. The two speak over port 50052 — see interfaces.

A client that crashes mid-call does not drop the caller. The engine carries on with the configuration it was given; what is lost is the client’s ability to steer and to record, not the conversation.

How it is shipped

As a container image, and only as one. There is no source build, no package to install and no plugin to load — everything the engine needs is inside it, including the compiled audio libraries.

What you supply is a port, a key, and a client to drive it. See deploy it.

What it does not carry

Audio never leaves the media path. The agent client receives transcripts and sends text; no audio frame crosses the control port.

That is what keeps a slow or distant client from being heard: the round trip it adds lands between turns, where tens of milliseconds are invisible — not inside the frame cadence, where they are a stutter the caller hears.

2.2 - Deploy it

Pull, run, verify, and connect a client.

Before you start

You need:

  • Docker, and access to the registry the image lives in.
  • A key for port 50052 — any 64-character random string. The agent client must present the same one.
  • An agent client to drive it. The agent server never places a call by itself; it waits to be told which room to join.

You do not need a GCP project, a service-account key, a model name or a prompt on this host. All of that arrives with each call.

1. Pull

docker login asia-south1-docker.pkg.dev
docker pull asia-south1-docker.pkg.dev/arctic-operand-415316/ulai/agent_server:v1.02

Pin the tag. latest makes it impossible to say afterwards what was running.

2. Run

docker run -d --name agent_server --restart unless-stopped \
  -p 127.0.0.1:50052:50052 \
  -p 8000:8000 \
  -e APP_GRPC_LISTEN_PORT=50052 \
  -e APP_HTTP_PORT=8000 \
  -e ULAI_GRPC_API_KEY='<64-character key>' \
  asia-south1-docker.pkg.dev/arctic-operand-415316/ulai/agent_server:v1.02

With docker compose

services:
  agent_server:
    image: asia-south1-docker.pkg.dev/arctic-operand-415316/ulai/agent_server:v1.02
    restart: unless-stopped
    ports:
      - "127.0.0.1:50052:50052"
      - "8000:8000"
    environment:
      APP_GRPC_LISTEN_PORT: 50052
      APP_HTTP_PORT: 8000
      ULAI_GRPC_API_KEY: "${ULAI_GRPC_API_KEY}"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 10s

3. Verify it started

curl -s http://localhost:8000/health

ok means the process is alive. It says nothing about calls — the server is healthy with zero calls running, which is its normal state.

Now read back what it thinks its configuration is:

docker logs agent_server 2>&1 | grep -A 25 "agent server configuration"
[env] ---- agent server configuration ----
[env]   APP_GRPC_LISTEN_PORT             50052   # gRPC port, all interfaces
[env]   APP_HTTP_PORT                    8000   # health only
[env]   ULAI_GRPC_API_KEY                set len=64 sha256:57a4cbf1   # demanded of callers; unset = NO AUTH
[env]   ONNXRUNTIME_LIB                  /usr/local/lib/libonnxruntime.so (file present)   # Smart Turn and Silero; the image sets it
[env] ---- end agent server configuration ----
[grpc] agentsession.v1 listening on [::]:50052 (AgentBridge served; AgentSession/AgentDispatch served)

Four things to confirm:

CheckWhy
ULAI_GRPC_API_KEY set len=64If it says (unset), anyone who reaches the port can control calls.
ONNXRUNTIME_LIB … (file present)The turn and voice-activity models cascade calls use are in place.
No NOT READ BY THIS PROCESS lineAnything listed there is set on this host and ignored — usually a rename, or a setting that now travels with the call.
[grpc] … listeningThe control port is up.

4. Point a client at it

Configure the agent client with this host’s address and the same key. In a typical client that is:

Client settingValue
Orchestrator addresshost:50052
Orchestrator API keythe same ULAI_GRPC_API_KEY
TLSon, if you put a proxy in front

Nothing appears in the agent server’s log until the client connects and starts a call. Silence here is normal.

5. Confirm a real call

Place one call through the client, then:

docker logs agent_server --tail 50

A healthy call looks like this:

[bridge] join: bridge=48c7ad… agent_id=d1a6b0… participant=agent_7f85…
[AIRoute] ai_project=my-project location=us-central1 creds=supplied (2347 bytes) (from the agent profile)
[bridge:48c7ad…] using the profile supplied with the request — no agent lookup, no database
[gemini] connected, greeting deferred (call=… model=… voice=Kore)
[bridge:48c7ad…] greeting primed

Two lines are the ones to know:

  • creds=supplied (N bytes) — the client sent credentials. If it says creds=engine default, the client sent none and the call will be refused.
  • greeting primed — the backend connected and the agent is about to speak. If this never appears, the call died before it could talk.

Upgrading

docker pull …/agent_server:v1.03
docker stop agent_server && docker rm agent_server
docker run -d … …/agent_server:v1.03      # same flags

Calls in flight during a restart end — there is no session migration. Restart when the log is quiet, or accept the dropped calls.

Next

2.3 - What a call carries

Why the environment is short, and what arrives per call instead.

The agent server is configured by two things, and only one of them is on this host.

2.3.1 - The agent profile

Everything about how a call runs, sent by the client.

Almost nothing about a conversation is configured on the agent server. The prompt, the voice, the GCP project, the service-account key, which behaviours are on — all of it arrives with each call, in something called the agent profile, sent by the agent client.

This matters to you as an operator for three reasons.

1. It explains the short environment list

You are not missing settings. There is no PROMPT, no VOICE, no GEMINI_MODEL to set here, because those are properties of an agent, not of a host. One agent server runs many different agents at once, each with its own.

2. It means one host can serve many customers

Because the GCP project and the service-account key travel with each call, a single agent server can run:

  • agent A, billed to customer A’s GCP project, on customer A’s key
  • agent B, billed to customer B’s, on customer B’s key

at the same time. As host configuration this would be impossible — one deployment could only ever serve one account, and serving two meant running two servers.

3. It means this host holds no credentials

There is no service-account key on this machine and no file to mount. If somebody copies the image, they get no customer’s credentials with it.

The trade is that a call carrying no credentials cannot run. There is nothing to fall back onto, so it is refused:

gemini: this call carried no credentials for project "my-project" —
the agent server holds none of its own, by design. Set google_credentials_json
on the agent, or GOOGLE_APPLICATION_CREDENTIALS on the agent client that sends
its profile

If you see that, the fix is on the client, not here. See troubleshooting.

What is in a profile

You do not set these — the client does — but knowing what a call carries makes the logs readable.

GroupExamples
The conversationprompt, greeting, closing line, voice, model, language
PlacementGCP project, region, service-account key, which AI vendor
Behaviourcan the caller interrupt, is the greeting interruptible, silence handling, answering-machine detection
Toolsthe functions this agent may call, answered by the client

Each field falls back on its own. A client sending a project but no region gets that project in the default region — not a half-configured call.

No environment variable on this host fills in a field the profile leaves out. The project and the key in particular must come from the call.

2.3.2 - Tools

Why an agent can look things up, and where that work happens.

An agent can do more than talk. Mid-call it can look up a balance, an order status, an appointment slot — anything the business it belongs to can answer.

None of that work happens on this host. The agent server has no database and no idea what any of those tools mean. It carries the request out to the agent client and carries the answer back.

model asks for "check_balance"
        │
        ▼
  agent server ──▶ agent client ──▶ the business's own system
        ◀──────────── answer ◀───────────────┘
        │
        ▼
model continues the conversation

What this means operationally

Tools are configured on the agent, not here. There is nothing to install, declare or permit on this host to make a tool available.

A slow tool is heard as silence. While the client is answering, the model is not speaking, and the caller hears that. The wait is bounded and always ends in an answer to the model — if the client does not answer in time, the model is told the lookup failed so it can apologise and carry on. A caller hearing “I couldn’t pull that up just now” is a far better outcome than a line that goes quiet.

So a tool that is slow on the client side shows up as pauses in conversations, not as errors in this log.

Two tools belong to the server. end_call — how a model hangs up, subject to a contract that checks it is entitled to — and a no-op tool declared beside it. The second exists so that a model which feels the urge to “use a tool” at a moment that is not an ending has somewhere harmless to put it, rather than reaching for end_call and cutting a live human off.

It is answered instantly, inside the server, precisely because sending it out and back would spend a multi-second silence avoiding a mistake that costs nothing.

In the logs

Tool activity appears against the call’s bridge id. A call that pauses oddly, with the agent going quiet and then apologising, is usually a tool the client was slow to answer — the place to look is the client’s logs, not this one.

2.3.3 - The shape of a call

From join to hangup, and who decides what along the way.

Joining

The room exists first. Whoever created it is the only party that knows which room the other participants are in, so the agent server never creates one — it is told which to join.

client ──JoinBridge──▶ server ──join──▶ control plane
                          │
                          ├──▶ SFU room     (WebSocket, Opus)
                          └──▶ AI backend   (the placement on the profile)

JoinBridge returns once the agent has a seat. The backend handshake continues after that.

Who speaks first

On a bridge the agent does. The room is already live, nobody rang, and an agent that joins and stays silent reads as a broken connection.

opening overrides this. Set it to caller when the agent is joining a conversation already in progress and should listen rather than announce itself.

During

The engine owns the timing. Every 20 ms of caller audio runs the chain — pre-clean, denoise, near-field foreground isolation, neural VAD — and the VAD is the authoritative turn driver.

The agent client receives transcripts and turn events and can steer between turns. What it cannot do is touch the media path: there is no way to send audio into the call and no way to change the audio processing. Timing stays with the engine.

Barge-in

When the caller speaks over the agent, the agent stops. What matters afterwards is not what the agent generated but what the caller actually heard, so a turn-ended event carries both:

An agent turn generated in full and heard for 300 ms did not happen, whatever the transcript says.

A client deciding “have they been told about the fee?” needs the second number, not the first.

Ending

Three ways a call ends:

The caller hangs upThe leg drops; the room reports it.
The client ends itOptionally running the closing sequence first.
The model calls end_callSubject to the end-call contract.

end_call is not a request the model gets for free. It must produce evidence — which of the legitimate endings this is, whether the closing question was actually spoken and answered — and that evidence is cross-checked against what the engine independently observed. A rejected end_call is answered with a re-prompt, so the agent keeps talking rather than going silent.

Every rejection is a prompt, and the engine is careful that rejections do not themselves become a source of repetition.

Detaching is not hanging up

If the client process dies mid-call, the engine carries on. The caller is not dropped because the record-keeper crashed; the cost is a reporting gap, which is the right one to take.

2.4 - Configuration

The short list that is environment, and the long list that is not.

Configuration is split by a single rule:

If it decides how a call runs, it travels with the call. If it decides how the process runs, it is an environment variable.

That split is why the environment list below is so short, and why the same binary can serve agents belonging to different customers at the same time. The per-call half is documented under the agent profile.

The process prints its whole configuration at boot, including which variables are set but not read — usually a rename, or a setting that has moved to the call:

[env] ---- agent server configuration ----
[env]   APP_GRPC_LISTEN_PORT             50052   # gRPC port, all interfaces
[env]   ULAI_GRPC_API_KEY                set len=64 sha256:57a4cbf1   # demanded of callers; unset = NO AUTH
[env]   ONNXRUNTIME_LIB                  /usr/local/lib/libonnxruntime.so (file present)   # Smart Turn and Silero; the image sets it
...
[env] ---- end agent server configuration ----

Process

This is the whole list. Nothing else in the environment is read.

VariableDefaultWhat it does
APP_GRPC_LISTEN_PORT50052The gRPC control surface. Binds all interfaces inside the container.
APP_HTTP_PORT8080/health, and nothing else.
ULAI_GRPC_API_KEY(none)Required as x-api-key on every RPC. Unset means no authentication.
PROMPT_LOG(on)Logs every text turn and tool result sent to the model. off stops it. Audio is never logged.
APP_PPROF_PORT6060pprof. Binds container-loopback, so publishing the port does nothing.
PROFILER_ENABLEDfalseStarts pprof at all.
ONNXRUNTIME_LIB(set by the image)Where ONNX Runtime is, for the turn and voice-activity models cascade calls use. Leave it alone.
GOMAXPROCS, GOMEMLIMIT, GOGC(auto)Runtime tuning. GOMEMLIMIT is derived from the cgroup limit when unset.

No longer read

Earlier images also read call settings from the environment. These are now ignored, because every call setting arrives with the call:

AI_BACKEND, AI_CLASSIFY_BACKEND, AI_TRANSCRIBE_MODEL, ULAI_LIVE_*, CASCADE_*, AMD_ENABLED, AUDIO_NOTHING, LOST_UTTERANCE_REPLAY, GEMINI_PROJECT_ID, GEMINI_LIVE_LOCATION, GOOGLE_APPLICATION_CREDENTIALS.

Remove them from your deployment. While they are set, the boot report lists the AI_*, GEMINI_*, GOOGLE_* and ULAI_* ones on its NOT READ BY THIS PROCESS line.

A call that arrives with no project or no key is refused with a message naming the field and both places it can be set. It is not quietly run against whatever ambient identity the host happens to carry — on a GCE instance that succeeds at authentication and then fails the Live handshake with insufficient authentication scopes, which describes neither the missing setting nor the account it borrowed.

2.5 - Interfaces

Two ports, and what talks to them.

The image exposes two ports and nothing else. There is no web UI, no admin endpoint and no configuration API — the agent server is driven entirely by the agent client.

PortProtocolWho connectsExpose it?
50052gRPCthe agent clientLoopback or private network only
8000HTTPyour monitoringAs needed

Port 8000 — health

One endpoint:

curl -s http://localhost:8000/health     # → ok

A liveness check, not a readiness one. It returns ok as soon as the process is up, including when no calls are running — which is the normal state. It does not tell you whether calls succeed; the log does that.

Safe to expose to a load balancer or monitoring system. It reveals nothing.

Port 50052 — control

This is where the agent client tells the server which room to join, which agent to run, and where to bill it. Everything about a call travels over this port.

It must be protected. Two reasons:

  1. There is no encryption. The server holds no TLS certificate. The API key and the service-account credentials each call carries cross this port in clear text.
  2. It controls live calls. A caller who reaches it can place agents into rooms and disconnect ones already talking.

The safe shapes are:

  • Bind to 127.0.0.1 and run a TLS-terminating proxy in front — what the published examples do.
  • Keep it on a private network that only the agent client can reach.

Publishing 0.0.0.0:50052 to the internet is not one of them.

Authentication

Every request must present ULAI_GRPC_API_KEY. The client is configured with the same value.

Leave the variable unset and the server accepts everything, including requests that disconnect live agents. It logs a warning saying so at startup, once:

[grpc] WARNING: ULAI_GRPC_API_KEY is unset — :50052 accepts unauthenticated
calls, including ones that disconnect live agents

One key, shared by every client. There is no per-client identity and no way to revoke one caller without rotating for all of them, so treat the key as an infrastructure secret rather than a per-team credential.

What it actually serves

For completeness, the port serves three gRPC services — AgentBridge (put an agent in a room, take it out), AgentSession (drive a live conversation) and AgentDispatch (receive calls as they start). Operating the server requires no knowledge of them; they matter to whoever builds the client.

Server reflection is enabled, so grpcurl can inspect the port for debugging — behind the same key:

grpcurl -H 'x-api-key: <key>' -plaintext localhost:50052 list

Outbound connections

The server also makes connections out, which your egress rules must allow:

ToFor
The Ulai control plane and SFU nodesjoining rooms and carrying audio
The AI backend (Vertex AI by default; also Cloud Speech-to-Text and Text-to-Speech on cascade calls)the conversation itself

Both are named per call by the client, not configured here.

2.6 - Operations

Deployment, observability and troubleshooting.

Running it, and finding out why it is not running.

2.6.1 - Deployment

The image, the ports, and the order things must be deployed in.

The image

It is a heavy image and has to be: it owns the audio. RNNoise is compiled from vendored C, TEN VAD is a prebuilt .so, and libopus, libsoxr and libspeexdsp are linked for the transport and the DSP chain. ONNX Runtime is included for the turn and voice-activity models that cascade calls use, and ONNXRUNTIME_LIB points at it.

The runtime stage is Debian slim and carries only the binary plus those shared libraries. It owns no business data — no database, no agent store, no credentials.

Ports

PortWhatExpose it?
50052gRPC (agentsession.v1)Loopback or private network only.
8000/healthAs needed.
6060pprofBinds container-loopback; publishing it does nothing.

Health

GET /health returns ok. It is a liveness check, not a readiness one — the process is healthy with zero calls running, which is the normal state.

Deployment order

The engine holds no project and no credentials, so the client that sends them must be deployed first. Deploying the server ahead of the client means every call is refused for want of a key.

The order:

  1. Deploy the client with its placement configured.
  2. Confirm [AIRoute] … creds=supplied (N bytes) on a call.
  3. Deploy the server.

Reversing it produces a working-looking server and a fleet of refused calls.

Sizing

One process handles many concurrent calls; the limit is CPU, and the audio chain is what spends it. Each call runs denoise, foreground isolation and neural VAD every 20 ms, and a cascade call runs its turn model on top.

GOMEMLIMIT is derived from the cgroup limit when unset — the boot log says what it picked:

[runtime] GOMEMLIMIT=13589MiB (/proc/meminfo limit=15987MiB, 85%)

Rolling restarts

Stop drains in-flight RPCs and then escalates to a hard stop after a timeout. One long-lived session stream must not hold a deploy open indefinitely — a session stream ends when its call does, which can be minutes away.

Calls in flight during a restart end. There is no session migration.

2.6.2 - Observability

The log lines worth knowing, and what they tell you.

Everything goes to stdout, so docker logs is the whole interface. Every line for a call carries its bridge id, which makes one call one filter:

docker logs agent_server --tail 100
docker logs agent_server 2>&1 | grep 48c7ad0bc30f     # one call
docker logs -f agent_server                            # follow

There are no metrics endpoints. Ship stdout to wherever you keep logs.

At boot

The whole configuration is printed, including variables that are set but not read — usually a rename, or a setting that now travels with the call:

[env] ---- agent server configuration ----
[env]   APP_GRPC_LISTEN_PORT             50052   # gRPC port, all interfaces
[env]   ULAI_GRPC_API_KEY                set len=64 sha256:57a4cbf1   # demanded of callers; unset = NO AUTH
[env]   ONNXRUNTIME_LIB                  /usr/local/lib/libonnxruntime.so (file present)   # Smart Turn and Silero; the image sets it
[env]   NOT READ BY THIS PROCESS: AI_BACKEND APP_ENV APP_NAME SERVICE_HTTP_PORT
[env]   (set in the environment but nothing here looks at them — usually a rename)
[env] ---- end agent server configuration ----

Secrets are shown as a length and a hash, never a value. A path that names a missing file says !! NO SUCH FILE.

Also at boot:

[grpc] agentsession.v1 listening on [::]:50052 (AgentBridge served; AgentSession/AgentDispatch served)

Per call

LineMeans
[bridge] join: bridge=… agent_id=… participant=…The agent has a seat.
[AIRoute] ai_project=… creds=supplied (N bytes) (from the agent profile)The profile carried a key. The one to check first.
using the profile supplied with the requestNo database lookup happened.
[gemini] connected, greeting deferredThe backend handshake succeeded.
ready — mode=… llm=…@… tts=…@… …A cascade call is up, with every setting it is running on.
greeting primedThe agent is about to speak.
roster: N other participant(s) in roomWho else is there.

[AIRoute] never prints a credential. It shows creds=supplied (2347 bytes) or creds=engine default — the latter meaning the profile carried none, which is a refusal.

Audio

When a call’s profile sets audio_nothing, the chain is cut down to the denoiser and the call’s log says so loudly:

[AUDIO] audio_nothing — pure RNNoise (preprocessor + foreground gate disabled)

That line in production is almost always a mistake — see troubleshooting.

Prompts

PROMPT_LOG is on unless set to off. Every text turn and tool result sent to the model is logged as an [→AI] line — the persona, the nudges, the closing instruction — which is the only record of what the model was actually told.

Turn it off if prompts or caller details should not reach your log store. Audio is never logged either way.

2.6.3 - Troubleshooting

Symptoms, and what actually causes them.

Start here

docker ps --filter name=agent_server           # is it running?
curl -s http://localhost:8000/health           # → ok
docker logs agent_server --tail 100            # what happened

Every line for a call carries its bridge id, so one call is one filter:

docker logs agent_server 2>&1 | grep 48c7ad0bc30f

The call joins, then dies immediately

Look for backend connect failed: right after joined session=….

this call carried no credentials for project "…"

The profile carried a project but no key. The engine holds none of its own, so there is nothing to fall back onto.

Confirm it in this server’s log — creds=engine default on the [AIRoute] line means the call carried no key:

docker logs agent_server 2>&1 | grep AIRoute | tail -5

The fix is on the agent client, which is the thing that must send a key. Nothing you change on this host will help: it holds no credentials by design.

this call carried no AI project

Same shape, other field. The profile named neither a project nor a key.

insufficient authentication scopes

Seen on older builds that fell back to ambient credentials. The host’s own identity — a GCE instance service account — authenticated successfully and then failed the Live handshake, because it lacks the cloud-platform scope Vertex needs.

Current images refuse the call before this can happen, with a message naming the missing field. If you see this, you are on an older image — check the tag with docker inspect --format '{{.Config.Image}}' agent_server.

DetectAnsweringMachine needs a Classifier

A call asked for answering-machine detection on an image that could not supply a classifier for it. Current images turn detection off rather than failing the call. Upgrade the image, or have the client stop asking for AMD.

The agent joins but never speaks

Check for greeting primed. If it is absent, the backend never connected — see above. If it is present and there is still silence, the caller’s side is the place to look: the room roster line says whether anyone else is actually there.

A setting on this host has no effect

You set AI_BACKEND, AMD_ENABLED, AUDIO_NOTHING or similar here, and calls behave as if you had not. Those are no longer read from the environment — see no longer read.

ONNXRUNTIME_LIB … !! NO SUCH FILE at boot

The image sets this to its own copy of ONNX Runtime, so this means it was overridden. Cascade calls still run, but without the turn and voice-activity models: they fall back to the built-in VAD and end turns on silence alone. Remove the override.

Calls are refused with unauthorized

The caller’s key does not match ULAI_GRPC_API_KEY. The boot log shows a hash of what the server expects:

[env]   ULAI_GRPC_API_KEY   set len=64 sha256:57a4cbf1

If it says (unset), the server is accepting everything, and the failure is elsewhere.

A tool never runs

Tools are configured on the agent and answered by the agent client — there is nothing to enable on this host. A tool that never runs is a client-side problem.

A call that pauses, goes quiet and then apologises is usually a tool the client answered too slowly. See tools.

Captions stop after the first utterance

Every caller utterance landing in one caption is a segment that never closed. Fixed in current images; on an older one the caller’s transcript accumulates into a single bubble while the agent’s lines appear separately. Upgrade.

The caller sounds gated or clipped

Have the client send audio_nothing on one call. It strips the chain to the denoiser alone, so if the problem disappears it is the pre-clean stage or the near-field foreground gate. There is no host-wide switch for this any more.

It is a diagnostic, not a setting. The full chain is the tuned one, and the stage it removes is what keeps a noisy room out of the model.