Deployment

This page describes where the platform runs and how a release reaches it. The pipeline covers the Bitbucket → Jenkins flow up to the point of a green build; this page picks up from there: what each host runs, the systemd units behind each service, and the deploy step that puts new code on the box.

There are no containers for the application services — Backend, Console, Voice fleet, and Dialler all run directly under systemd, with code synced from Bitbucket and dependencies installed in place. The only Docker on the platform is the LiveKit stack.

Host roles

The platform is deployed by role, not by machine. A small deployment co-locates several roles on one host; a busy one splits them out and adds more fleet hosts. The roles, and what runs on each, are:

How Jenkins deploys to a host

Every role follows the same shape once the build is green: Jenkins reaches the target host, checks out the release, installs dependencies in place, and restarts the relevant systemd unit. The only differences are the build command (uv sync for Python, npm for the Console) and which unit gets restarted.

API / App host

The control-plane host. The Backend API serves admin and runtime-config APIs on :8080; the Console is a static build served by nginx; MongoDB and Redis back the Backend. Small deployments co-locate all four.

What runs

# scripts/run.sh — what the unit executes
uvicorn voxbridge.app:app \
  --host 0.0.0.0 \
  --port 8080 \
  --app-dir src
# host/port from VOXBRIDGE_HOST (0.0.0.0) / VOXBRIDGE_PORT (8080)

# Built once per release, output is static dist/ served by nginx
npm ci
npm run build        # = tsc -b && vite build
# Ori's brand .env is committed in the repo; the build copies it to root .env.
# nginx serves dist/ and proxies /api/* to the Backend on :8080.

systemd unit (Backend)

# /etc/systemd/system/voxbridge.service
[Unit]
Description=Backend API
After=network.target mongod.service redis-server.service

[Service]
Type=simple
WorkingDirectory=/opt/voxbridge
EnvironmentFile=/opt/voxbridge/.env
ExecStart=/opt/voxbridge/scripts/run.sh
Restart=always

[Install]
WantedBy=multi-user.target

Jenkins deploy step

uv sync
sudo systemctl restart voxbridge
curl -fsS http://127.0.0.1:8080/health

npm ci
npm run build
# publish dist/ to the nginx web root

Fleet host(s)

The runtime host. Each fleet host runs N single-worker uvicorn processes, one per call slot, each bound to its own Unix socket. nginx fans incoming calls across the sockets with least_conn and max_conns=1, so each worker handles exactly one call at a time. Recordings are uploaded to a MinIO instance local to the host.

The worker model

This is the part that does not look like an ordinary web service. Instead of one uvicorn process with many workers, the fleet uses one systemd instance per worker, each pinned to its own socket. That makes a single call the unit of failure and lets you restart one worker without touching the others.

# scripts/run_worker.sh — one worker on one socket
uv run uvicorn voxcore.app:app \
  --uds "$VOXCORE_SOCKET" \
  --workers 1 \
  --app-dir src

Templated systemd unit

A single template unit, voxcore@.service, is enabled once per worker as voxcore@1, voxcore@2, … voxcore@N. The instance number %i selects the socket.

# /etc/systemd/system/voxcore@.service
[Unit]
Description=Voice fleet worker %i
After=network.target

[Service]
Type=simple
WorkingDirectory=/opt/voxcore
EnvironmentFile=/opt/voxcore/.env
Environment=VOXCORE_SOCKET=/tmp/voxcore_%i.sock
ExecStart=/opt/voxcore/scripts/run_worker.sh
Restart=always

[Install]
WantedBy=multi-user.target

# Enable N workers
for i in $(seq 1 16); do sudo systemctl enable --now "voxcore@$i"; done

nginx in front of the sockets

nginx terminates public traffic and balances across the worker sockets. Two details are non-negotiable, both shipped in the repo template at infra/nginx/voxcore-fleet.conf.template:

• map $http_upgrade for the Connection header. A hardcoded Connection "upgrade" works for WebSocket calls but breaks the HTTP POST routes (dialout returns 422).
• 429-retry to the next upstream for the short POST routes (/attach, /livekit/dialout, /livekit/widget). These return before the call pipeline finishes, so least_conn can route a later request to a worker that is logically busy; retrying a 429 lets nginx try the next free socket.

# map: pick the right Connection header per request type
map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

upstream voxcore_workers {
    least_conn;
    server unix:/tmp/voxcore_1.sock  max_conns=1 fail_timeout=10;
    server unix:/tmp/voxcore_2.sock  max_conns=1 fail_timeout=10;
    # … one line per worker …
    server unix:/tmp/voxcore_16.sock max_conns=1 fail_timeout=10;
}

server {
    listen 443 ssl;

    # WebSocket call entry
    location /ws/ {
        proxy_pass http://voxcore_workers;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
    }

    # Short POST routes — retry 429 to the next worker
    location = /attach {
        proxy_pass http://voxcore_workers;
        proxy_next_upstream error timeout http_429 non_idempotent;
        proxy_next_upstream_tries 16;
    }
}

Jenkins deploy step

uv sync

sudo systemctl restart 'voxcore@*'
# or, one worker at a time:
sudo systemctl restart voxcore@2

curl -fsS https://fleet.example.com/health/fleet

Scaling the fleet

SIP / LiveKit host

The telephony host. It runs the LiveKit media server and the livekit-sip bridge as a docker-compose stack (the only Docker on the platform), and it hosts the single Dialler instance.

LiveKit

LiveKit and livekit-sip are brought up with the docker-compose stack from the repo (infra/livekit/setup.sh). It holds the SIP trunks per carrier, needs use_external_ip on cloud hosts, and needs a webhook configured for inbound SIP dispatch. LiveKit is the one component that is not redeployed by the app pipeline — it is brought up once and left running.

# Brought up once on the SIP host
docker compose up -d   # livekit + livekit-sip

Dialler

The Dialler is a background asyncio worker (with a small health server), not a web app. It runs from /opt/voxdialler and ticks every LOOP_INTERVAL_SECONDS (2.0): query running campaigns → predictive pacing → fair-share → lease calls atomically in MongoDB → place SIP calls via LiveKit → on answer, attach to a free fleet worker through POST /attach.

# /opt/voxdialler/deploy/voxdialler.service
[Unit]
Description=Dialler
After=network.target

[Service]
Type=simple
WorkingDirectory=/opt/voxdialler
EnvironmentFile=/opt/voxdialler/.env
ExecStart=/opt/voxdialler/scripts/run.sh   # uv run python -m voxdialler.main
Restart=always

[Install]
WantedBy=multi-user.target

Jenkins deploy step

uv sync

sudo systemctl restart voxdialler

curl -fsS http://127.0.0.1:8090/health
curl -fsS http://127.0.0.1:8090/metrics

WSS ingress host (optional)

When more than one fleet host serves a single public domain, an HAProxy ingress fronts them. Carrier and CRM WSS traffic hits HAProxy, which spreads it across the fleet hosts; each fleet host still runs its own nginx and worker sockets unchanged. The templates live in the fleet repo under infra/haproxy/ (render.sh, add-fleet.sh, haproxy.cfg.template). Adding a fleet to the ingress is config-only — render the config with the new backend and reload HAProxy; no fleet-host code changes.

A release, end to end

# Illustrative bitbucket-pipelines.yml — build + test on push, deploy from main.
# (Snippet only; not committed to the repos.)
image: python:3.12

pipelines:
  branches:
    main:
      - step:
          name: Install, lint, test
          script:
            - pip install uv
            - uv sync
            - uv run pytest -v
      - step:
          name: Deploy
          deployment: production
          trigger: manual
          script:
            - ssh deploy@fleet-host 'cd /opt/voxcore && git pull && uv sync && sudo systemctl restart "voxcore@*"'

// Illustrative Jenkinsfile (declarative) — same flow, Jenkins-driven.
// (Snippet only; not committed to the repos.)
pipeline {
  agent any
  stages {
    stage('Install & test') {
      steps {
        sh 'uv sync'
        sh 'uv run pytest -v'      // Console: npm ci && npm run lint && npm run build && npm test
      }
    }
    stage('Deploy') {
      when { branch 'main' }
      steps {
        sh '''
          ssh deploy@$TARGET_HOST "cd $APP_DIR && git checkout $RELEASE_TAG && uv sync"
          ssh deploy@$TARGET_HOST "sudo systemctl restart $UNIT"
        '''
      }
    }
    stage('Health check') {
      steps {
        sh 'curl -fsS $HEALTH_URL'  // /health, /health/fleet, or dialler :8090/health
      }
    }
  }
}

Where to go next