e21a5241bf
Switches the production backing store from SQLite to PostgreSQL 16, per DR-2.
The migration trigger is two concurrent writers on the my-deepagent ORM
tables — which first appears with FastAPI (M8-Py). Doing the cut now keeps
the surface area small while M8-Py is still planning.
Production deps: `asyncpg`, `psycopg[binary]`, `langgraph-checkpoint-postgres`.
Test deps: `aiosqlite` (the bulk of unit + integration tests stay on sqlite
tmp_path for speed; the E2E suite and the new checkpointer tests exercise
the live Postgres path).
Highlights
- `persistence/db.py`: dialect-aware connect listener. SQLite still gets
WAL + busy_timeout=5000 + foreign_keys=ON; Postgres gets `SET TIME ZONE 'UTC'`.
Added `Database.dialect_name` + `drop_schema` (test-only).
- `persistence/checkpointer.py`: SqliteSaver → AsyncPostgresSaver. API is
now async (`async with`) and takes a connection string. SQLAlchemy URL
prefixes (`+asyncpg`, `+psycopg`) are auto-stripped to a plain libpq DSN
(`_to_psycopg_dsn` helper, 4 unit tests).
- `persistence/upsert.py` (new): `insert_for(session)` — dialect-aware UPSERT
helper. Picks `postgresql.insert` or `sqlite.insert` based on the bound
engine. Replaces 5 hardcoded `sqlite_insert` call sites in `budget.py`,
`recovery.py`, `cli/doctor.py`.
- `persistence/models.py`: `RunRow` partial unique index declares both
`postgresql_where=` and `sqlite_where=` for cross-dialect correctness.
- `config.py`: default `database_url` now
`postgresql+asyncpg://devflow:devflow@localhost:55432/mydeepagent`. v3
`devflow` DB preserved untouched; v4 lives in a fresh `mydeepagent` DB.
- `cli/doctor.py` check 8: dialect-aware DB liveness probe. Postgres path
runs `SELECT 1` (pg_isready equivalent); SQLite keeps `PRAGMA integrity_check`.
- `alembic/env.py`: env-aware URL resolution (`MYDEEPAGENT_DATABASE_URL` >
`DATABASE_URL` > default). Async driver prefixes are mapped to the sync
equivalents alembic needs.
- `alembic/versions/9f2a6c79667e_v0_2_baseline_schema_postgres.py` (new):
fresh baseline autogenerated against live Postgres. Old SQLite migrations
(`79945fdc2649`, `839f2233e346`) deleted — v0.2 starts a clean history.
- `tests/conftest.py` (new): `pg_db_url` async fixture creates a fresh DB
per test against docker-compose `devflow-postgres` and drops it on
teardown after terminating lingering backends.
- `tests/integration/test_checkpointer.py`: rewritten for AsyncPostgresSaver
(4 pure DSN-converter unit tests + 3 async context-manager integration tests).
- `tests/integration/test_e2e_workflow.py`: switched to `pg_db_url`. Real
OpenRouter E2E now exercises the production Postgres path end-to-end.
Recovery
- Previous SQLite database at the platformdirs data_dir is NOT auto-migrated;
v0.1.0 was the only release that wrote to it. Set
`MYDEEPAGENT_DATABASE_URL=sqlite+aiosqlite:///<path>` to read it.
- The v3 `devflow` Postgres DB is preserved untouched (separate database
name); to inspect: `psql -h localhost -p 55432 -U devflow -d devflow`.
Gates
- ruff check + ruff format --check + mypy --strict: PASS (102 source files)
- pytest non-E2E: 576 PASS (5.46 s)
- pytest E2E real OpenRouter on Postgres: 1 PASS (122.93 s, ~$0.05/run)
--no-verify: lefthook still TS-only (deleted in 0e61b2d but still queryable
in git history).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
81 lines
2.7 KiB
Python
81 lines
2.7 KiB
Python
"""Test fixtures shared across unit + integration tests.
|
|
|
|
v0.2 PR #1: tests run against the live Postgres container managed by
|
|
docker-compose. Each test that needs DB isolation requests the
|
|
``pg_db_url`` fixture, which creates a fresh database per test and drops it
|
|
on teardown.
|
|
|
|
Prerequisites:
|
|
docker compose up -d postgres # devflow-postgres on 55432
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import os
|
|
import uuid
|
|
from collections.abc import AsyncIterator
|
|
from typing import Final
|
|
|
|
import psycopg
|
|
import pytest_asyncio
|
|
|
|
# Maintenance connection — used only to CREATE DATABASE / DROP DATABASE.
|
|
# `postgres` is the bootstrap DB present on every Postgres install.
|
|
_MAINTENANCE_DSN: Final[str] = os.environ.get(
|
|
"MYDEEPAGENT_TEST_MAINTENANCE_DSN",
|
|
"postgresql://devflow:devflow@localhost:55432/postgres",
|
|
)
|
|
|
|
|
|
def _async_url(db_name: str) -> str:
|
|
"""Return the SQLAlchemy + asyncpg URL for *db_name*."""
|
|
return f"postgresql+asyncpg://devflow:devflow@localhost:55432/{db_name}"
|
|
|
|
|
|
def _create_test_database() -> str:
|
|
"""Create a fresh test database with a random suffix and return its name."""
|
|
db_name = f"test_{uuid.uuid4().hex[:16]}"
|
|
# autocommit=True is required for CREATE DATABASE (cannot run in a tx block).
|
|
with psycopg.connect(_MAINTENANCE_DSN, autocommit=True) as conn:
|
|
with conn.cursor() as cur:
|
|
cur.execute(f'CREATE DATABASE "{db_name}"')
|
|
return db_name
|
|
|
|
|
|
def _drop_test_database(db_name: str) -> None:
|
|
"""Forcefully terminate connections and drop *db_name*. Idempotent."""
|
|
with psycopg.connect(_MAINTENANCE_DSN, autocommit=True) as conn:
|
|
with conn.cursor() as cur:
|
|
# Kick any lingering connections held by aiosqlite-style pools that
|
|
# didn't dispose cleanly. WITH (FORCE) is Postgres 13+.
|
|
cur.execute(
|
|
"""
|
|
SELECT pg_terminate_backend(pid)
|
|
FROM pg_stat_activity
|
|
WHERE datname = %s AND pid <> pg_backend_pid()
|
|
""",
|
|
(db_name,),
|
|
)
|
|
cur.execute(f'DROP DATABASE IF EXISTS "{db_name}"')
|
|
|
|
|
|
@pytest_asyncio.fixture
|
|
async def pg_db_url() -> AsyncIterator[str]:
|
|
"""Yield an isolated Postgres database URL; drop the DB on teardown.
|
|
|
|
Usage::
|
|
|
|
async def test_something(pg_db_url: str) -> None:
|
|
db = Database(pg_db_url)
|
|
await db.init_schema()
|
|
...
|
|
|
|
The returned URL uses the asyncpg driver. To get a sync URL for tools
|
|
like alembic, replace ``postgresql+asyncpg://`` with ``postgresql+psycopg://``.
|
|
"""
|
|
db_name = _create_test_database()
|
|
try:
|
|
yield _async_url(db_name)
|
|
finally:
|
|
_drop_test_database(db_name)
|