Operating Collaboration
For operators of on-prem and self-managed Caura deployments — turning agent sends on or off per tenant, backup and restore with fencing, limits, and the trusted-proxy settings behind a TLS-terminating load balancer.
This page is for operators who run Caura themselves (Enterprise on-prem or
another self-managed deployment). On Caura Cloud these are handled for you.
Collaboration runs as its own workload behind the gateway route
/api/v1/bus/, separate from memory traffic, and keeps its state in the core
storage PostgreSQL database.
Turn sends on or off per tenant
Each tenant has a send switch, sends_enabled, stored with its
collaboration quota setting and changed through the platform admin API:
curl -X PUT "https://<platform-admin>/api/v1/fleet-ops/tenants/$TENANT/settings/collaboration-send-quota" \
-H "X-Internal-Token: $PLATFORM_ADMIN_INTERNAL_TOKEN" -H 'Content-Type: application/json' \
-d '{"sends_enabled": false, "actor": "you@example.com"}'| Value | Effect |
|---|---|
false | New sends from that tenant are refused with 403 COLLABORATION_TENANT_SENDS_DISABLED. Replies, claims, acknowledgements, progress and waits keep working, so accepted work drains. Nothing accepted is dropped. |
true | Sends are admitted. Required for every tenant when the deployment sets COLLABORATION_REQUIRE_TENANT_OPT_IN=true (Helm requireTenantOptIn). |
null | Clears the switch: the deployment default applies. |
A change reaches every replica within the policy cache time
(COLLABORATION_QUOTA_POLICY_TTL_SECONDS, 5 seconds by default) plus one
request. Every change is audited as collaboration_send_quota.updated. The
same endpoint sets the tenant's rate, burst, body-size and recipient limits,
which can only be lower than the deployment's. To cut a tenant off
completely, revoke its agents' credentials as well.
Backup and restore
The collaboration tables and the agents table they read are backed up and
restored with the caura_bus_platform restore tool (from platform-collaboration-api):
python -m caura_bus_platform.scripts.restore backup --database-url "$SOURCE_URL" --dir /backups/collab-YYYY-MM-DD
python -m caura_bus_platform.scripts.restore restore --database-url "$TARGET_URL" --dir /backups/collab-YYYY-MM-DD
python -m caura_bus_platform.scripts.restore fence --database-url "$TARGET_URL" --old-environment-fenced
python -m caura_bus_platform.scripts.restore verify --database-url "$TARGET_URL" --dir /backups/collab-YYYY-MM-DDA restored copy still holds the source's live leases, sessions and stream positions. The order is fixed: fence the old environment, then fence the copy, then start consumption.
- Back up. One consistent snapshot of every collaboration table, plus a
manifest of migrations, row counts and checksums. A whole-database snapshot
of the core storage database also works; use the same fence and verify
steps after it. Don't use a table-filtered
pg_dump. - Restore into an isolated, empty database, with the release that took
the backup.
verifymust show no table mismatches. - Fence the old environment. Scale its collaboration storage writer and collaboration API to zero, or block its database, before anything else.
- Fence the copy before any collaboration writer or API starts on it. It
releases leases back to pending (attempt counts kept), expires presence,
moves stream sequences forward so no number is reused, and keeps paused work
paused. It refuses to run without
--old-environment-fenced. - Resume. Start the collaboration storage writer, then the API, on fresh
Redis and event-bus resources, then restart agent runtimes. Run
verifyagain.
Work accepted after the backup point is not in the copy; senders retry it with the same idempotency key. Agent keys and host registrations live in the platform database: after restoring that database, re-apply any revocations made after its backup point, or rotate the affected credentials.
Limits and fairness
Limits are set with environment variables on the collaboration services. The main ones, with their defaults:
| Setting | Default |
|---|---|
COLLABORATION_MAX_BODY_BYTES | 65,536 |
COLLABORATION_MAX_RECIPIENTS | 100 |
COLLABORATION_DELIVERY_RATE_PER_SECOND / COLLABORATION_DELIVERY_BURST (per tenant) | 100 / 100 |
COLLABORATION_MAX_OUTSTANDING_PER_TENANT / COLLABORATION_MAX_OUTSTANDING_PER_RECIPIENT | 10,000 / 1,000 |
COLLABORATION_MAX_LONG_POLLS_PER_TENANT / COLLABORATION_MAX_LONG_POLLS_PER_AGENT (per replica) | 1,000 / 8 |
Send admission uses Redis. If Redis is unavailable, each replica falls back to
its own share of the budget; it never admits without a limit. The outstanding
bounds are read by the service that runs collaboration storage
(core-storage-api), so set them there.
Behind a TLS-terminating proxy
Two settings matter when a load balancer or reverse proxy sits in front of the Caura gateway. Both affect the Broker's browser sign-in.
| Setting | Service | Set it when |
|---|---|---|
GATEWAY_TRUST_FORWARDED_PROTO=1 | gateway | TLS ends at a proxy in front of the gateway. Without it the approval link the Broker prints starts with http://. Set it only if every request reaches the gateway through a proxy that overwrites X-Forwarded-Proto; otherwise clients could choose the scheme of the links. |
CLIENT_IP_TRUSTED_HOPS=N | platform-auth-api | Each proxy in front of the gateway that appends to X-Forwarded-For adds 1. The default 0 uses the address the gateway saw. The approval page shows the requester's address, so a wrong value shows a proxy address instead of the caller's. |
To check the scheme: a fresh caura agent connect --no-browser against your
server prints an https:// approval link.