Skip to content

Incident: Python 3.10 Compatibility & Missing Migrations

Date: 2025-12-22 Severity: High (Production APIs down, missing database tables) Duration: Unknown (discovered during routine check) Affected: ky04, mi20-clevenger tenant APIs

Summary

Production tenant APIs (ky04, mi20-clevenger) had two issues:

  1. Python Compatibility: Code using datetime.UTC (Python 3.11+) was deployed to tenants running Python 3.10 venvs, causing crash loops
  2. Missing Tables: Tenants were missing 13 database tables because the MIGRATIONS list in upgrade.py was outdated

Timeline

Time Event
2025-12-22 12:40 v0.4.46 deployed to ky04 and mi20-clevenger
2025-12-22 15:10 Issue discovered during routine tenant check
2025-12-22 15:15 Root cause #1 identified: ImportError: cannot import name 'UTC' from 'datetime'
2025-12-22 15:18 Python fix committed (v0.4.47)
2025-12-22 15:20 v0.4.47 deployed, services restored
2025-12-22 18:40 Root cause #2 identified: tenants missing 13 tables (15 vs 28)
2025-12-22 18:45 Migrations run manually on both tenants
2025-12-22 18:50 MIGRATIONS list updated in upgrade.py

Root Cause

The src/api/auth.py file was updated to use datetime.UTC, which was added in Python 3.11:

from datetime import UTC  # Python 3.11+ only!

expire = datetime.now(tz=UTC)

This passed testing on the development environment (Python 3.12.9) but failed on production tenant venvs (Python 3.10.12).

Environment Comparison

Environment Python Version Result
cbapp dev (nominates) 3.12.9 Works
testsite 3.12.9 Works
ky04 3.10.12 Broken
mi20-clevenger 3.10.12 Broken

Resolution

Changed to use the backwards-compatible timezone.utc:

from datetime import timezone

expire = datetime.now(tz=timezone.utc)

Root Cause #2: Missing Migrations

The MIGRATIONS list in cbtenant/api/routes/upgrade.py was outdated:

Before (6 scripts): 

MIGRATIONS = [
    "scripts/create_schema.py",
    "scripts/create_default_users.py",
    "scripts/add_person_columns.py",
    "scripts/migrate_add_import_metadata.py",
    "scripts/add_person_indexes.py",
    "scripts/add_segment_tables.py",
]

Missing scripts (4): - add_segment_survey_table.py - add_work_queue_tables.py - add_patch_tables.py - add_integration_settings_table.py

This caused tenants to have only 15 tables instead of 28.

Lessons Learned

  1. Dev/Prod parity: Development environment (Python 3.12) didn't match production (Python 3.10)
  2. No version guard: Code used Python 3.11+ features without checking compatibility
  3. Testing gap: No CI testing against Python 3.10
  4. Migration tracking: New migration scripts in cbapp must be added to cbtenant's MIGRATIONS list

Action Items

  1. Short-term: Upgrade all tenant venvs to Python 3.12 (see GitHub issue)
  2. Long-term: Add CI matrix testing for Python 3.10 and 3.12
  3. Process: Document minimum Python version in CLAUDE.md and requirements.txt
  4. Process: When adding migrations to cbapp, also update cbtenant's MIGRATIONS list