Test Coverage Analyzer¶

Script: scripts/update_coverage.py

A tool for analyzing test coverage, identifying gaps, and generating test stubs - like Alembic for tests.

Quick Start¶

# See what's missing
python scripts/update_coverage.py

# Generate test stubs (preview)
python scripts/update_coverage.py --generate --dry-run

# Generate with AI assistance
python scripts/update_coverage.py --generate --ai

# Full update: tests + screenshots + docs
python scripts/update_coverage.py --full --ai

Features¶

1. API Coverage Analysis¶

Scans src/api/routes/*.py to discover all endpoints, then compares against existing tests in tests/test_*.py.

$ python scripts/update_coverage.py

Discovering API endpoints...
  Found 150 endpoints
Discovering existing tests...
  Found 344 API tests

Coverage Gaps:
  API: 71 endpoints missing tests

2. UI Page Coverage¶

Scans pages from scripts/capture_screenshots.py and compares against E2E tests in tests/e2e/.

Coverage Gaps:
  UI: 4 pages missing tests

3. Test Stub Generation¶

Generate basic test stubs for missing coverage:

# Dry-run to preview
python scripts/update_coverage.py --generate --dry-run

# With AI assistance (uses Anthropic API or Claude Code)
python scripts/update_coverage.py --generate --ai

4. AI-Powered Test Generation¶

The script supports two AI modes:

Claude Code Mode: When running within Claude Code, prompts are displayed for Claude to process interactively.

Anthropic API Mode: When running standalone with ANTHROPIC_API_KEY set, uses the API directly to generate tests.

# In Claude Code
python scripts/update_coverage.py --generate --ai
# Outputs prompts for Claude to process

# Standalone with API key
export ANTHROPIC_API_KEY=sk-ant-...
python scripts/update_coverage.py --generate --ai
# Calls API and generates test code

5. Screenshot Updates¶

Update screenshots for missing UI pages:

python scripts/update_coverage.py --screenshots

6. Documentation Updates¶

Generate a coverage report in docs/TEST_COVERAGE.md:

python scripts/update_coverage.py --docs

Command Line Options¶

Option	Description
`--base-url URL`	Test site URL (default: https://mi20.nominate.ai)
`--generate`	Generate test stubs for missing coverage
`--ai`	Use AI to generate better test implementations
`--dry-run`	Show what would be generated without writing
`--screenshots`	Update screenshots for missing pages
`--docs`	Update documentation with coverage report
`--full`	Run full update: tests + screenshots + docs
`--json`	Output results as JSON (for scripting)
`--verbose, -v`	Verbose output with all gaps listed

Prompt Templates¶

AI prompts are stored in scripts/prompts/:

api_test.md - Template for API endpoint tests
ui_test.md - Template for UI/E2E page tests

Templates use placeholders like {method}, {path}, {docstring} that are filled from endpoint/page metadata.

JSON Output¶

For programmatic use:

python scripts/update_coverage.py --json > coverage.json

{
  "endpoints": 150,
  "pages": 22,
  "api_tests": 344,
  "ui_tests": 107,
  "gaps": {
    "api": [{"method": "GET", "path": "/endpoint", ...}],
    "ui": [{"path": "/page", "name": "page_name", ...}]
  },
  "ai_mode": false,
  "base_url": "https://mi20.nominate.ai"
}

Integration with CI¶

Add to your CI pipeline to fail builds with missing coverage:

- name: Check test coverage
  run: python scripts/update_coverage.py
  # Exits with code 1 if gaps found

How It Works¶

API Discovery: Parses route files with regex to find @router.get/post/etc decorators
Test Discovery: Scans test files for client.get/post/etc calls to determine what's tested
Coverage Matching: Normalizes paths and methods to compare endpoints vs tests
Gap Identification: Lists untested endpoints and pages
Stub Generation: Creates basic test skeletons or uses AI for smarter tests

Examples¶

Find all untested endpoints¶

python scripts/update_coverage.py -v

Generate stubs for communications module only¶

python scripts/update_coverage.py --generate --dry-run 2>&1 | grep -A10 "communications"

Full AI-assisted update¶

export ANTHROPIC_API_KEY=sk-ant-...
python scripts/update_coverage.py --full --ai

This will: 1. Generate AI-powered test stubs for missing endpoints 2. Update screenshots for missing pages 3. Update docs/TEST_COVERAGE.md with current status