dev-puppeteer/my-deepagent/CHANGELOG.md

# Changelog

## [Unreleased]

### Added
- **v0.2 PR #1 — Postgres migration**: production backing store switched from
  SQLite to PostgreSQL 16 ahead of M8-Py (FastAPI) per DR-2.
  - `pyproject.toml`: `asyncpg>=0.30` + `psycopg[binary]>=3.2` +
    `langgraph-checkpoint-postgres>=2.0.0` added to runtime; `aiosqlite>=0.20`
    moved to `[dependency-groups].dev` (test-only); `langgraph-checkpoint-sqlite`
    removed.
  - `src/my_deepagent/persistence/db.py`: dialect-aware connect listener —
    SQLite still gets `WAL` + `busy_timeout=5000` + `foreign_keys=ON`, Postgres
    gets `SET TIME ZONE 'UTC'`. New `Database.dialect_name` property + `drop_schema`
    method for tests.
  - `src/my_deepagent/persistence/checkpointer.py`: `SqliteSaver` →
    `AsyncPostgresSaver`. API is now async (`async with`) and takes a
    connection string; SQLAlchemy URL prefixes (`postgresql+asyncpg://`,
    `postgresql+psycopg://`) are auto-stripped to a plain libpq DSN. New
    `_to_psycopg_dsn` helper covered by 4 unit tests.
  - `src/my_deepagent/persistence/upsert.py` (new): `insert_for(session)` —
    dialect-aware UPSERT helper. Picks `postgresql.insert` or `sqlite.insert`
    based on the bound engine's dialect. Replaces 5 hardcoded `sqlite_insert`
    call sites in `budget.py`, `recovery.py`, and `cli/doctor.py`.
  - `src/my_deepagent/config.py`: `database_url` default switched from
    `sqlite+aiosqlite:///<data_dir>/database.sqlite3` to
    `postgresql+asyncpg://devflow:devflow@localhost:55432/mydeepagent`. The v3
    `devflow` DB is preserved untouched; v4 lives in a fresh `mydeepagent` DB.
  - `src/my_deepagent/persistence/models.py`: `RunRow.__table_args__` partial
    unique index now declares **both** `postgresql_where=` and `sqlite_where=`
    so the index is partial on both dialects.
  - `src/my_deepagent/cli/doctor.py`: check 8 (`disk+db`) is now dialect-aware
    — Postgres path runs `SELECT 1` (pg_isready equivalent: proves
    reachability + auth + DB exists); SQLite path keeps
    `PRAGMA integrity_check`. Doctor docstring updated.
  - `alembic/env.py`: env-aware URL resolution — `MYDEEPAGENT_DATABASE_URL` >
    `DATABASE_URL` > Postgres default. Async driver prefixes
    (`+asyncpg`, `+aiosqlite`) are mapped to the sync equivalents alembic
    needs (`+psycopg`, plain `sqlite`).
  - `alembic/versions/9f2a6c79667e_v0_2_baseline_schema_postgres.py` (new):
    fresh baseline autogenerated against live Postgres. Old SQLite baseline
    `79945fdc2649` + partial-index migration `839f2233e346` deleted.
  - `tests/conftest.py` (new): `pg_db_url` async fixture. Creates a fresh
    Postgres database per test (against docker-compose `devflow-postgres`)
    via the maintenance `postgres` DB; drops on teardown after terminating
    any lingering backends. Used by the E2E suite and the new checkpointer
    tests.
  - `tests/integration/test_checkpointer.py`: rewritten for AsyncPostgresSaver
    (4 pure DSN-converter tests + 3 async context-manager tests).
  - `tests/integration/test_e2e_workflow.py`: switched from `sqlite+aiosqlite`
    tmp_path to `pg_db_url`. Real OpenRouter E2E now exercises the production
    Postgres path end-to-end (~122 s, ~$0.05/run).

### Migration trigger (per DR-2)
- The bound is *two concurrent writers* on `runs` / `run_phases` / `llm_calls`.
  Today the CLI is the only writer — but M8-Py (FastAPI) introduces a second
  one, and SQLite WAL allows only a single concurrent writer. Doing the move
  *before* M8-Py lands gives the test surface time to harden.
- Recovery: previous SQLite database at
  `~/Library/Application Support/my-deepagent/database.sqlite3` (macOS) /
  `$XDG_DATA_HOME/my-deepagent/database.sqlite3` is **not migrated** —
  v0.1.0 was the only release that wrote to it and v0.2 starts a fresh
  history. Set `MYDEEPAGENT_DATABASE_URL=sqlite+aiosqlite:///<path>` to
  read the legacy file if needed.

### Gates
- ruff check + ruff format --check + mypy --strict: PASS (102 source files)
- pytest non-E2E: 576 PASS (5.46 s) — bulk on sqlite tmp_path, new
  checkpointer suite on Postgres `pg_db_url`
- pytest E2E real OpenRouter: 1 PASS (122.93 s) on Postgres backend


## [0.1.0] - 2026-05-16

First tagged milestone of the Python rewrite. The pre-Python-rewrite TypeScript
monorepo has been removed (commit `0e61b2d`); recovery is available via the
`pre-python-rewrite` tag at `c9fed71`.

### Added
- Step 15 — End-to-end real OpenRouter integration: `tests/integration/test_e2e_workflow.py`
  runs `spec-and-review@1` workflow (spec → review → verify) end-to-end against real
  OpenRouter DeepSeek in ~76s for ~$0.05 per run. `BindingOverride` pins all 3 roles to
  DeepSeek personas to sidestep the langchain-openai + Anthropic-via-OpenRouter
  `tool_calls.args` JSON-string ValidationError (known v0.1.0 limit). New seed personas:
  `openrouter-deepseek-spec-writer@1` (capabilities: spec_write, phase_planning;
  max_cost_per_call_usd=0.01) and `openrouter-deepseek-code-reviewer@1` (capabilities:
  code_review, evidence_check; max_cost_per_call_usd=0.01). Persona count test updated
  to 12. `WorkflowEngine._build_envelope` now inlines the artifact JSON Schema directly
  in the prompt so the LLM sees exact required fields. `WorkflowEngine._record_llm_call`
  fills every NOT NULL `LlmCallRow` column (thread_id, persona_version, role, turn_index,
  cached_tokens, reasoning_tokens, cost_usd_input/output, etc.). `CostMiddleware` now
  probes both `usage_metadata` and `response_metadata.token_usage` (prompt_tokens /
  completion_tokens fallback) to capture OpenAI-compatible streamed responses forwarded
  by OpenRouter.
- Step 12 — Doctor full 8-check + OpenRouter pricing fetch: `mydeepagent doctor`
  now runs 8 checks (python / uv / git / workspace_root / config+governance /
  openrouter_api_key / openrouter_ping + pricing upsert / disk+sqlite integrity).
  `mydeepagent pricing` lists the cached OpenRouter pricing matrix from the
  persisted `model_pricing` table. `mydeepagent run` preview now reads from the
  persisted `model_pricing` table when populated, falling back to the static seed
  otherwise. 26 new tests (23 unit + 3 integration).
- Step 11 — Audit log + secret scrubbing: append-only `{state_dir}/audit.jsonl`
  recording every tool call (name/args/duration/error). `AuditToolMiddleware` now
  ships with a built-in JSONL recorder (`file_recorder`), attached automatically in
  `WorkflowEngine` and Interactive REPL. `structlog` configured project-wide via
  `my_deepagent.logging.configure_logging`, with a `_scrub_processor` that redacts
  OpenRouter / Anthropic / OpenAI / LangSmith / GitHub / GitLab API keys plus
  generic Bearer tokens before they reach stderr or JSON sinks. `audit.py` provides
  `append_audit_record` (O_APPEND, 0o600 permissions), `read_audit_records` (with
  optional limit, corrupt-line skip), and `make_audit_recorder` async factory.
  19 new tests (8 audit unit, 9 logging unit, 3 audit-middleware integration).
- Step 10 — Interactive REPL: `mydeepagent` (no subcommand) launches a prompt_toolkit
  REPL with `--agent` / `--model` overrides, slash commands (`/help`, `/quit`, `/exit`,
  `/agent`, `/model`, `/clear`, `/stats`, `/budget`, `/runs`), file refs
  (`@path/to/file.py` expansion with repo-root containment check), and
  `CostMiddleware`-wired agent calls so spending is metered per interactive session.
  `slash.py` implements `parse_slash` + `SlashRegistry`. `CostMiddleware` gains
  `interactive_session_id` parameter. 21 new tests (10 slash unit, 5 file-ref unit,
  3 CLI integration, 3 updated CLI unit).
- Step 9 — Crash recovery + concurrency: `sweep_orphan_runs(db)` in
  `my_deepagent.recovery` marks non-terminal runs/phases as failed at app startup so
  active-run uniqueness slots (partial unique index `ux_active_run_repo_base`) are freed;
  `mydeepagent runs list/show/resume` CLI in `my_deepagent.cli.runs` (list with optional
  `--state` filter, show by full UUID or 6+ char prefix, resume stub with exit-2 hint);
  SIGTERM/SIGINT graceful shutdown in `WorkflowEngine` (`install_signal_handlers`,
  `_on_signal`, `_force_cancel_inflight`; 30s grace then cancel in-flight tasks);
  auto-sweep on `mydeepagent run` before any new phase begins. 21 new tests.
- Step 8 — Budget guardrails: `BudgetTracker` (SQLite WAL ledger via `BudgetLedgerRow`,
  on_hit policy block/warn_continue/prompt, per-run + per-day + per-persona-daily
  scopes) in `my_deepagent.budget`; cost preview before `mydeepagent run` (rich table
  with per-phase est.) via `my_deepagent.monitoring.cost_estimator`;
  `CostMiddleware` integrated with `BudgetTracker` (pre-call assert + post-call record);
  `WorkflowEngine` accepts optional `budget_tracker` and `pricing` kwargs (backward-
  compatible); CLI: `mydeepagent budget` (ledger), `mydeepagent stats --by model|persona|day`,
  `mydeepagent costs` (alias); `--no-preview` flag on `mydeepagent run`.
  28 new tests.
- Step 7 — Workflow engine: `WorkflowEngine` in `my_deepagent.engine` orchestrates
  phase loop, artifact watcher (write_file/edit_file detection), jsonschema validation
  with one repair retry, approval gate, and final report compose (JSON + Markdown).
  `ArtifactWatcherMiddleware` in `my_deepagent.middleware.artifact_watcher` intercepts
  write_file/edit_file tool calls targeting the expected artifact path.
  `RunEventType` + `run_idempotency_key` in `my_deepagent.run_event` (closed event set,
  deterministic idempotency keys per plan v2.0 §13.1).
  `cli/run.py` exposes `mydeepagent run <workflow.yaml>`.
  `tui/approval.py` prompts the user for approve/reject/request_changes/abort.
  FK-safe persistence: WorkflowTemplateRow and AgentPersonaRow upserted before RunRow
  to satisfy SQLite FK ordering constraints.
  18 new tests: 12 engine unit/integration tests + 6 artifact watcher tests.
- Step 6 — Distribution: `mydeepagent init/login/logout/keys/doctor` CLI commands;
  platformdirs-based data dirs; OS keyring (macOS Keychain / Linux Secret Service /
  Windows Credential Store) for API keys via `my_deepagent.keys`; first-run
  governance consent in `governance.py`; secret resolution priority
  (config → env → keyring → error) in `my_deepagent.secrets`; i18n catalog
  (ko / en) under `my_deepagent.i18n` controlled by `MYDEEPAGENT_LANG`.
- persistence/models.py (P0-1): partial unique index `ux_active_run_repo_base` on `runs(repo_path, base_branch) WHERE state NOT IN ('completed','failed','aborted')` — prevents duplicate active runs per repo/branch
- persistence/models.py (P0-3): FK constraints added to `RunRow.template_id` (RESTRICT), `RunBindingRow.persona_id` (RESTRICT), `InteractiveSessionRow.persona_id` (RESTRICT), `RunEventRow.phase_id` (CASCADE), `ApprovalRequestRow.phase_id` (CASCADE), `ArtifactRow.phase_id` (CASCADE), `ToolCallRow.run_id/phase_id/interactive_session_id` (CASCADE), `LlmCallRow.run_id/phase_id/interactive_session_id` (CASCADE), `PhaseFeedbackRow.run_id/phase_id` (CASCADE)
- alembic/versions/839f2233e346: new migration adding partial unique index and all FK constraints above; uses SQLite table-rebuild pattern with PRAGMA foreign_keys=OFF/ON guard
- persistence/checkpointer.py (P0-4): removed `get_checkpointer` (leaking connection helper); only `get_checkpointer_ctx` context manager is now exported
- tests/integration/test_checkpointer.py: 5 tests for checkpointer ctx lifecycle (file creation, parent dir, connection cleanup, lock-free concurrent use)
- tests/integration/test_persistence.py: 7 new P0 verification tests (active-run partial index blocks/allows, cascade-delete of phase_feedback+run_phases, RESTRICT on template delete, index exists in sqlite_master)
- tests/unit/test_session.py: full rewrite to deepagents dataclass API — FilesystemPermission attribute access (.mode/.paths/.operations), build_backend type dispatch (5 cases), _map_operations deduplication (8 cases), _spec_to_permission mapping, updated _subagent_to_dict and _resolve_openrouter_api_key tests; 47 unit tests total
- tests/integration/test_openrouter_smoke.py: real OpenRouter/DeepSeek smoke test (3 tests, ~$0.001-$0.003/run, max_tokens=50); skipped automatically when no API key is configured; validates ChatOpenAI response, usage_metadata tokens, and deepagents CompiledStateGraph end-to-end
- pyproject.toml: registered `integration` pytest marker to silence --strict-markers error
- v0.1.0 scaffolding (Step 0): src/tests/docs trees, ruff/mypy/pre-commit/alembic config
- Seed assets copied to docs/schemas/ (personas/workflows/artifacts validated)
- Core module (Step 1): config, enums, errors, hash + unit tests
- Persona / Workflow / Binding module (Step 2): pydantic schemas, YAML loaders, deterministic auto-select, override, consent store with atomic write
- Step 1 review patches (P0/P1): exception chain context suppression, classmethod LSP fix, workspace_root realpath canonicalization, config_invalid error mapping

### Changed
- deepagents 0.6.1 LocalShellBackend + permissions conflict workaround: removed `permissions` block from all 10 seed personas; `SafetyShellMiddleware` now enforces destructive-command + secret-path policy at the tool layer for local_shell backend agents.
- `build_agent` automatically prepends `SafetyShellMiddleware` to every agent and skips `permissions` kwarg when `deepagents_backend == "local_shell"`.
- `SafetyShellMiddleware` extended with secret-path enforcement: `read_file`/`write_file`/`edit_file`/`ls` tool calls are blocked when `file_path`/`path` matches any `DENY_PATH_PATTERNS` glob (wcmatch GLOBSTAR|IGNORECASE|DOTGLOB).
- All env vars require `MYDEEPAGENT_` prefix (e.g. `MYDEEPAGENT_OPENROUTER_API_KEY`, `MYDEEPAGENT_BUDGET_DAILY_USD`). `.env.example` updated accordingly. This isolates my-deepagent's env namespace from other tools.
- Persona / Workflow / FilesystemPermission models now store list-valued fields as tuples (deep immutability — prevents post-construction mutation that would invalidate compute_hash()).

### Known limitations (v0.1.0)
- `usage_metadata` is sometimes empty for responses forwarded by OpenRouter (deepagents
  wraps the underlying ChatOpenAI response so token counts may not surface). The
  `CostMiddleware` recorder still fires and a `LlmCallRow` row is persisted, but
  `input_tokens` / `output_tokens` may read as 0 — the E2E test treats this as a known
  limit. v0.2 will probe more response shapes (raw chunks / callbacks).
- Anthropic models via OpenRouter currently fail with a `tool_calls.args` JSON-string
  vs dict ValidationError inside `langchain-openai`. Workaround: pin DeepSeek personas
  via `BindingOverride`. Tracking for v0.2.
- `mydeepagent runs resume <run_id>` is a stub (exit-2 hint only); workflow replay
  from a half-run state is not yet implemented.