Operations

• 1: Deployment
• 2: Networking
• 3: Observability
• 4: Troubleshooting

1 - Deployment

The gateway is distributed as a container image. A deployment is one container on a host with a routable public IP, run with host networking.

docker run

docker run -d --name ulai-sip \
  -e ULAI_CONTROL_PLANE_URL=https://stgcp.ulai.co.in \
  -e SIP_ROUTING_REDIS_URL='redis://USER:PASSWORD@redis.ulai.co.in:6379' \
  -e SIP_PUBLIC_IP=148.113.58.51 \
  -e SIP_LISTEN_ADDR=0.0.0.0:5060 \
  -e SIP_TRANSPORT=udp \
  -e SIP_RTP_HOST=0.0.0.0 \
  -e SIP_RTP_PORT_LOW=10000 \
  -e SIP_RTP_PORT_HIGH=10500 \
  -e SIP_HTTP_PORT=8082 \
  --network host \
  asia-south1-docker.pkg.dev/arctic-operand-415316/ulai/sip:v6

Three things in that command are load-bearing:

• --network host — see networking. Publishing the RTP range with -p spawns hundreds of userland proxies and rewrites the INVITE’s source address, which the trunk ACL then rejects with 403.
• SIP_PUBLIC_IP — the address the carrier is told to send media to. It must be the host’s own routable address, and never 0.0.0.0.
• SIP_RTP_HOST=0.0.0.0 — what the RTP sockets bind to. On a cloud VM the public IP usually is not on any interface, so binding to it fails outright.

Pass secrets with --env-file rather than -e if the host’s process list is readable:

docker run -d --name ulai-sip --network host \
  --env-file /etc/ulai/sip.env \
  asia-south1-docker.pkg.dev/arctic-operand-415316/ulai/sip:v6

docker compose

services:
  sip-gateway:
    image: asia-south1-docker.pkg.dev/arctic-operand-415316/ulai/sip:v6
    container_name: ulai-sip
    network_mode: host
    env_file:
      - .env
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "./health-check.sh"]
      interval: 30s
      timeout: 5s
      start_period: 15s
      retries: 3

network_mode: host replaces a ports: block — the container binds the host’s ports directly, which is the only arrangement that keeps the carrier’s source address intact and the RTP range usable.

docker compose up -d
docker compose logs -f

What the image contains

A Debian slim runtime with the gateway binary, libopus, ca-certificates, and curl plus sipsak for the health check. It runs as an unprivileged user (sipgw, uid 10001) — SIP 5060 and the RTP range are unprivileged ports, so nothing here needs root.

Exposed ports:

Upgrading

Pull the new tag, then replace the container. There is no state on disk — every call is in memory and every routing decision is read fresh — so a replacement is a restart, not a migration:

docker pull asia-south1-docker.pkg.dev/arctic-operand-415316/ulai/sip:v7
docker stop --time 70 ulai-sip
docker rm ulai-sip
docker run -d --name ulai-sip --network host --env-file /etc/ulai/sip.env \
  asia-south1-docker.pkg.dev/arctic-operand-415316/ulai/sip:v7

Live calls do not survive the replacement beyond the drain window below, so prefer a quiet period — or run a second host and move the trunk across.

Restart and shutdown

SIGINT/SIGTERM starts a drain rather than a kill:

shutdown signal received — draining for up to 1m0s

New calls are refused (503 on both legs), in-flight calls get up to 60 seconds to finish, and anything still up is then cancelled with a further 5 seconds of grace. The routing-store connection is closed last, so a draining call’s CALL_HANGUP still has somewhere to go.

Give the container a matching stop timeout so Docker does not SIGKILL through the drain:

docker stop --time 70 ulai-sip

2 - Networking

Ports

The RTP range sizes the gateway’s concurrency: roughly 250 simultaneous calls at the default range, since allocation uses even ports only by RTCP convention. Widen SIP_RTP_PORT_LOW/HIGH to carry more.

The carrier must be able to reach SIP_PUBLIC_IP on the SIP port and the whole RTP range. A firewall that allows 5060 but not 10000–10500 produces a call that connects and then has no audio — the single most common misconfiguration.

Port publishing versus host networking

Run the container with --network host.

Publishing the ports with -p breaks the gateway in two separate ways:

1. 501 userland proxies. -p 10000-10500:10000-10500/udp spawns one docker-proxy process per port. That is slow to start, heavy at rest, and adds a hop to every media packet.
2. The trunk ACL rejects the call. A proxied INVITE arrives from the bridge gateway’s address, not the carrier’s. The resolver checks the source IP against the trunk’s ACL, does not find it, and answers 403 Forbidden.

The same reasoning is why SIP_PUBLIC_IP must be a routable address: it is copied into every SDP and Contact the gateway sends, and it is where the carrier will stream media.

Bind address versus advertised address

Two different variables, routinely confused:

On AWS, GCP, Azure and most VPS providers the public IP is attached at a NAT or load-balancer layer and is not present on any interface inside the VM. Binding to it fails with cannot assign requested address, so the default 0.0.0.0 is the right answer nearly everywhere.

NAT and ICE

The room side is WebRTC, so it needs ICE. The gateway is STUN-only by default (stun:stun.l.google.com:19302), which works on a public host and fails behind symmetric NAT. Add TURN there:

-e TURN_URLS='turn:turn.example.com:3478?transport=udp,turns:turn.example.com:5349' \
-e TURN_USERNAME=ulai \
-e TURN_PASSWORD=...

The startup line reports how many ICE servers were configured:

sip-sfu-gateway: HTTP listening on :8082 (control plane https://stgcp.ulai.co.in, ice_servers=1)

ice_servers=1 means STUN only.

Symmetric RTP

Carriers behind NAT routinely send RTP from a port they never advertised in SDP. The gateway latches onto the source of the first well-formed packet and sends there, rather than trusting the SDP address — without which roughly half of inbound calls would be one-way. See media.

Transport choice

A TLS trunk calling back a UDP-only listener is a real failure mode: the BYE never arrives and the call would hang forever, which is what SIP_RTP_TIMEOUT_SECONDS exists to catch.

3 - Observability

Reading the log

Every line of a call is prefixed with the number and the leg it arrived on, so a busy gateway’s log can be read one call at a time:

[+919363399639] [sip-originate] call answered — call_id=3e114ad4…, joining room 527f78a3…
[+919363399639] [sfugw] webrtc state: connected
[+919363399639] [sfugw] DTMF: 5
[+919363399639] [sip-originate] call finished (session 527f78a3…, call_id 3e114ad4…)

grep on the number gives one call; grep on the sip_call_id joins the gateway’s log to carrier CDRs and to the published telecom events.

Health checks

There are two legs, and /health only covers one:

curl -fsS http://127.0.0.1:8082/health
sipsak -s sip:healthcheck@127.0.0.1:5060

health-check.sh in the repo runs both and exits non-zero if either fails. It is what docker-compose.yml and the CI verification step use. Only one HEALTHCHECK can be active per image, which is why both checks live in one script.

/health returns 200 whenever the process is serving. It does not assert that the routing store is reachable — that is reported at startup and again on every call that fails because of it.

Live calls

watch -n2 'curl -s localhost:8082/calls | jq'

The list is sorted oldest-first, so rows do not reshuffle between polls. It is also the number the drain reports:

drain deadline reached with 3 call(s) still up — cancelling them

The control-plane wire log

Every request the gateway makes to the control plane — create, join, terminate — is logged with its headers and body:

[control-plane] → POST https://stgcp.ulai.co.in/api/v1/sessions headers={Authorization: Bearer ulai…kzM (51 chars), Content-Type: application/json, X-System-Secret: ahd8…3dg (26 chars)} body={"max_participants":8,…}
[control-plane] ← POST /api/v1/sessions 201 Created (128ms)

It exists because “what did we actually send?” is a question best answered by the log, not by reading the SDK — which matters most when the answer is a 403 and the question is which header was missing.

Three rules it follows:

• Credentials are masked, never printed whole: first and last four characters plus the length. Enough to tell which secret went out without putting it in every log shipper that reads this process’s output.
• Auth headers are always reported, present or not. For an auth failure, Authorization: <not sent> is the single most useful thing a log can say.
• Only failure bodies are logged. A success body carries the join ticket and events token, which are credentials in their own right; a failure body is the control plane’s explanation.

Events

CALL_ANSWERED and CALL_HANGUP are published to the platform’s telecom stream for every answered call, carrying the routing decision, the room, the numbers and — on hangup — duration_seconds. They are the record to build dashboards and alerting on; the gateway keeps no history of its own beyond the live table. See routing.

Startup lines worth alerting on

routing store reachable
WARNING: routing store unreachable (...) — every call will be rejected until it recovers

The gateway deliberately starts either way: a store that is down now may be up a second from now, and a SIP listener answering an honest 500 is worth more than one that is not there at all.

4 - Troubleshooting

The gateway will not start

sip-sfu-gateway: ULAI_CONTROL_PLANE_URL is required (e.g. https://stgcp.ulai.co.in)
SIP_PUBLIC_IP must be a routable address, not 0.0.0.0 — it is advertised to the carrier

Configuration is validated once, and every problem is reported together — fix them all, then restart. 0.0.0.0 in SIP_PUBLIC_IP is rejected on purpose: it is not a listen address, it is what the carrier is told to stream media to.

Other startup failures:

Calls are rejected

The SIP status tells you which check failed:

Every rejection is logged with the reason immediately before it:

[+919363399639] [sip-gw] rejecting 403: source IP not allowed by the trunk ACL: …

The call connects but there is no audio

Almost always the media path, not the signalling path:

1. The RTP range is not open. 5060 allowed, 10000–10500 blocked, is the classic. Check the firewall and the security group.
2. SIP_PUBLIC_IP is wrong. The carrier is streaming to whatever the SDP said. Check it against the host’s actual routable address.
3. Port publishing instead of host networking. Media arrives at the proxy rather than the process.
4. One-way only? That is usually NAT on the carrier’s side. The gateway already latches onto the observed source address; if it is still one-way, the outbound direction is being dropped upstream.

Outbound calls fail

If the 202 comes back but the phone never rings, the failure is in the background half and only the log has it:

[+919363399639] [sip-originate] Originate failed: …
[+919363399639] [sip-originate] RATE LIMITED by trunk … (SIP 429, Retry-After 30s) — this gateway does not pace originations; the caller must slow down

The room created for that call is terminated rather than left running.

488 Secure media required

A carrier with secure trunking enabled answered a plain RTP/AVP offer. The gateway infers SRTP from the trunk’s transport — TLS trunks offer it, UDP and TCP do not — so either fix the stored transport, or force it per request with "offer_srtp": true. See routing.

404 from the trunk on a self-hosted SBC

A dialplan matching literal digit patterns will not match a leading +. Send "dial_verbatim": true. Hosted ITSPs want the opposite — E.164 with the plus.

A call never ends

If the far end’s BYE is lost — a TLS trunk calling back a UDP-only listener never reaches us at all — the call and its room would stay up until the process exits. SIP_RTP_TIMEOUT_SECONDS (default 30) is the backstop: a call whose inbound audio stops for that long is ended as if the far end had hung up. It is paused while the call is on hold. Setting it to 0 disables the backstop entirely, which is rarely what you want.

The agent talks to a ringtone

It should not — the gateway joins the room only after the callee answers, precisely so a participant appearing in the roster is a reliable cue. If it happens, something else joined the room early; the gateway’s own join is logged as call answered — … joining room ….

The agent keeps talking after the caller hangs up

The session is not being terminated with the call. A room the gateway created is always torn down; a room it was pointed at (X-Session-Id, or session_id on originate) is torn down too unless SIP_TERMINATE_SESSION_ON_HANGUP=false. Check that variable first.

Duplicate agents on one call

A mid-call re-INVITE handled as a new call produces exactly this — a second agent, a second model session, a second billing row. In-dialog INVITEs are handled separately for that reason. If you see it, capture the SIP flow and check whether the second invocation followed a re-INVITE from the carrier’s SBC.