Tenant Deployment Guide¶

Single Source of Truth for CampaignBrain Tenant Deployments

Last Updated: 2026-01-06

Table of Contents¶

Current Tenant Status
Quick Reference
Provisioning Checklist
Critical Lessons Learned
Environment Configuration
Service Dependencies
Open Issues & Roadmap
Validation & Monitoring
Troubleshooting

Current Tenant Status¶

Tenant	Domain	Ports	Status	Last Validated
testsite	testsite.nominate.ai	32300/32301	Active	2026-01-06
mi20-clevenger	mi20.nominate.ai	32310/32311	Active	2026-01-06
ky04	ky04.nominate.ai	32320/32321	Active	2026-01-06
zach-lahn-for-governor	lfg.nominate.ai	32350/32351	Active	2026-01-06

Port Allocation¶

Ports are allocated in increments of 10:

Range	Tenant
32300-32309	testsite
32310-32319	mi20-clevenger
32320-32329	ky04
32330-32339	(reserved)
32340-32349	(reserved)
32350-32359	zach-lahn-for-governor
32360-32369	(next tenant)

Quick Reference¶

Essential Commands¶

# Validate all tenants
python scripts/validate_tenant_env.py

# Validate specific tenant
python scripts/validate_tenant_env.py --tenant testsite --fix

# Check service status
sudo systemctl status {tenant}-api {tenant}-frontend

# View logs
sudo journalctl -u {tenant}-api -f

# Restart services
sudo systemctl restart {tenant}-api {tenant}-frontend

# Test health endpoint
curl https://{domain}/api/health

File Locations¶

Purpose	Path
Tenant root	`/home/bisenbek/projects/nominate/{tenant}/`
Environment	`/home/bisenbek/projects/nominate/{tenant}/.env`
Database	`/home/bisenbek/projects/nominate/{tenant}/db/pocket.db`
Nginx config	`/etc/nginx/sites-available/{domain}.conf`
Systemd services	`/etc/systemd/system/{tenant}-api.service`
SSL certs	`/etc/letsencrypt/live/{domain}/`

Provisioning Checklist¶

Full 22-step checklist: TENANT-PROVISIONING-CHECKLIST.md

Critical Steps (Don't Skip These!)¶

1. Clone & Setup¶

TENANT_NAME="newtenant"
git clone git@github.com:Nominate-AI/cbapp.git /home/bisenbek/projects/nominate/$TENANT_NAME
cd /home/bisenbek/projects/nominate/$TENANT_NAME
python3 -m venv venv
./venv/bin/pip install -r requirements.txt

2. Environment Configuration¶

cp .env.example .env
# Edit with tenant-specific values (see Environment Configuration below)

3. Database Initialization¶

mkdir -p db
python scripts/create_schema.py
python scripts/create_admin_user.py --username admin --password 'SECURE_PASSWORD'

4. Systemd Services¶

Create API service: /etc/systemd/system/${TENANT_NAME}-api.service Create Frontend service: /etc/systemd/system/${TENANT_NAME}-frontend.service

sudo systemctl daemon-reload
sudo systemctl enable ${TENANT_NAME}-api ${TENANT_NAME}-frontend
sudo systemctl start ${TENANT_NAME}-api ${TENANT_NAME}-frontend

5. Nginx Configuration¶

Create /etc/nginx/sites-available/{domain}.conf

sudo ln -sf /etc/nginx/sites-available/{domain}.conf /etc/nginx/sites-enabled/
sudo nginx -t && sudo systemctl reload nginx

6. SSL Certificate¶

sudo certbot --nginx -d {domain}

7. Local DNS Resolution (CRITICAL!)¶

# Add to /etc/hosts - THIS IS REQUIRED!
echo "127.0.0.1    {domain}" | sudo tee -a /etc/hosts

8. Service API Keys¶

Generate keys for YASP, CBFiles, and add Anthropic key - see Service Dependencies

9. Validation¶

python scripts/validate_tenant_env.py --tenant ${TENANT_NAME}
curl https://{domain}/api/health

Critical Lessons Learned¶

1. /etc/hosts Entry is REQUIRED¶

Problem: Server cannot connect to itself via public IP (NAT hairpinning failure).

Symptoms: - First API request works, subsequent requests timeout (~35 seconds) - E2E tests fail with connection timeouts - Health checks from server to itself fail

Solution: Add domain to /etc/hosts:

127.0.0.1    testsite.nominate.ai
127.0.0.1    mi20.nominate.ai
127.0.0.1    ky04.nominate.ai
127.0.0.1    lfg.nominate.ai

Discovery: Found 2026-01-06 when lfg.nominate.ai tests failed. Issue #68.

2. Use DB_PATH, Not DATABASE_URL¶

Problem: Legacy DATABASE_URL=sqlite:///db/pocket.db format is deprecated.

Solution: Use DB_PATH=db/pocket.db (no protocol prefix).

Migrated: mi20-clevenger, ky04 (2026-01-06)

3. All Service Keys Must Be Present¶

Required external service keys: - YASP_API_KEY - Survey integration - FILES_API_KEY + FILES_TENANT_BUCKET - File storage - ANTHROPIC_API_KEY - AI chat features - API_KEYS - Service-to-service auth - CBMODELS_BASE_URL + CBMODELS_TENANT_ID - Campaign data API

Validation: Run python scripts/validate_tenant_env.py --fix to check for missing keys.

4. Placeholder Logos Required¶

If custom logos aren't available, copy defaults:

cp src/app/static/images/defaults/logo-*.svg src/app/static/images/

Environment Configuration¶

Required Variables¶

Variable	Example	Notes
`APP_NAME`	"Smith for Senate"	Display name
`PROJECT_NAME`	"smith2026"	Internal ID (lowercase)
`FRONTEND_PORT`	32300	Unique per tenant
`BACKEND_PORT`	32301	Unique per tenant
`SECRET_KEY`	(generate)	JWT signing key
`FRONTEND_URL`	"https://smith.nominate.ai"	Public URL
`BACKEND_API_URL`	"http://localhost:32301/api"	Internal API URL
`DB_PATH`	"db/pocket.db"	Database file
`API_KEYS`	(generate)	Service-to-service auth

Generate Secrets¶

# SECRET_KEY (32 bytes hex)
python -c "import secrets; print(secrets.token_hex(32))"

# API_KEYS (16 bytes hex)
python -c "import secrets; print(secrets.token_hex(16))"

Full Reference¶

See ENV_CONFIGURATION.md for complete variable documentation.

Service Dependencies¶

YASP Survey Platform¶

# Generate API key (on YASP server)
cd /home/bisenbek/projects/nominate/cbsurveys
python -m yasp.cli create-key --user-email admin@{domain} --key-name {tenant}

Add to .env:

YASP_BASE_URL=https://surveys.nominate.ai/api/v1
YASP_API_KEY=yasp_xxx

CBFiles Storage¶

Add to .env:

FILES_BASE_URL=https://files.nominate.ai/api/v1
FILES_API_KEY=cbfiles_xxx
FILES_TENANT_BUCKET={tenant}

CBModels Campaign Data API¶

Add to .env:

CBMODELS_BASE_URL=http://localhost:32411
CBMODELS_TENANT_ID={tenant}

Anthropic (Shared Key)¶

ANTHROPIC_API_KEY=sk-ant-xxx

See DEPLOYMENT-SERVICES-SETUP.md for detailed setup instructions.

Open Issues & Roadmap¶

Active Issues¶

#	Title	Priority	Status
#68	E2E test failures (session timeouts)	High	Open
#66	Automated tenant health checks	Medium	Open
#67	Centralized secret management	Low	Open
#45	Python 3.10 → 3.12 upgrade	Low	Open
#46	CBFiles/MinIO setup	Low	Open

Recently Closed¶

#	Title	Resolution
#65	Tenant Alignment Audit	All items resolved 2026-01-06

Recommended Improvements¶

Health Check Automation (#66)
Cron job to check all tenants every 5 minutes
Alert on service failures
Dashboard for tenant status
Centralized Secrets (#67)
Move API keys to HashiCorp Vault or similar
Eliminate per-tenant .env management
Enable key rotation
Tenant Provisioning Script
Single command to provision new tenant
Automated key generation
Database initialization

Validation & Monitoring¶

Validation Script¶

# Validate all tenants
python scripts/validate_tenant_env.py

# Sample output:
# ✅ testsite
#    Path: /home/bisenbek/projects/nominate/testsite
#    Version: v0.4.118
#    Services: API=active, Frontend=active

Health Checks¶

# Quick health check
for domain in testsite.nominate.ai mi20.nominate.ai ky04.nominate.ai lfg.nominate.ai; do
  echo -n "$domain: "
  curl -s -o /dev/null -w "%{http_code}\n" https://$domain/api/health
done

Log Monitoring¶

# Follow API logs
sudo journalctl -u {tenant}-api -f

# Recent errors
sudo journalctl -u {tenant}-api --since "1 hour ago" -p err

# Nginx access logs
sudo tail -f /var/log/nginx/{domain}.access.log

Troubleshooting¶

Service Won't Start¶

# Check logs
sudo journalctl -u {tenant}-api --no-pager -n 50

# Common issues:
# - Wrong Python path in service file
# - Missing .env file
# - Port already in use
# - Missing dependencies (run pip install -r requirements.txt)

502 Bad Gateway¶

# Check if backend is running
curl http://localhost:{backend_port}/api/health

# Check nginx config
sudo nginx -t

# Restart nginx
sudo systemctl reload nginx

API Requests Timeout (35+ seconds)¶

Cause: Missing /etc/hosts entry (NAT hairpinning failure)

Fix:

echo "127.0.0.1    {domain}" | sudo tee -a /etc/hosts

Authentication Errors¶

# Check SECRET_KEY is set
grep SECRET_KEY /home/bisenbek/projects/nominate/{tenant}/.env

# Regenerate if needed
python -c "import secrets; print('SECRET_KEY=' + secrets.token_hex(32))"

Survey Integration Not Working¶

# Verify YASP key
grep YASP_API_KEY /home/bisenbek/projects/nominate/{tenant}/.env

# Test YASP connection
curl -H "X-API-Key: {yasp_key}" https://surveys.nominate.ai/api/v1/health

Document	Purpose
TENANT-PROVISIONING-CHECKLIST.md	Step-by-step deployment
ENV_CONFIGURATION.md	Environment variables reference
DEPLOYMENT-SERVICES-SETUP.md	External service configuration
hygiene/FINDINGS-2026-01-06.md	Audit report

Contacts¶

Role	Contact
Infrastructure	[Team contact]
YASP Admin	[Team contact]
CBFiles Admin	[Team contact]

This document consolidates tenant deployment knowledge. Update after any significant deployment changes.

Tenant Deployment Guide¶

Table of Contents¶

Current Tenant Status¶

Port Allocation¶

Quick Reference¶

Essential Commands¶

File Locations¶

Provisioning Checklist¶

Critical Steps (Don't Skip These!)¶

1. Clone & Setup¶

2. Environment Configuration¶

3. Database Initialization¶

4. Systemd Services¶

5. Nginx Configuration¶

6. SSL Certificate¶

7. Local DNS Resolution (CRITICAL!)¶

8. Service API Keys¶

9. Validation¶

Critical Lessons Learned¶

1. /etc/hosts Entry is REQUIRED¶

2. Use DB_PATH, Not DATABASE_URL¶

3. All Service Keys Must Be Present¶

4. Placeholder Logos Required¶

Environment Configuration¶

Required Variables¶

Generate Secrets¶

Full Reference¶

Service Dependencies¶

YASP Survey Platform¶

CBFiles Storage¶

CBModels Campaign Data API¶

Anthropic (Shared Key)¶

Open Issues & Roadmap¶

Active Issues¶

Recently Closed¶

Recommended Improvements¶

Validation & Monitoring¶

Validation Script¶

Health Checks¶

Log Monitoring¶

Troubleshooting¶

Service Won't Start¶

502 Bad Gateway¶

API Requests Timeout (35+ seconds)¶

Authentication Errors¶

Survey Integration Not Working¶

Related Documentation¶

Contacts¶