dev-puppeteer/my-deepagent/CHANGELOG.md

# Changelog

## [Unreleased]

### Added
- Step 15 — End-to-end real OpenRouter integration: `tests/integration/test_e2e_workflow.py`
  runs `spec-and-review@1` workflow (spec → review → verify) end-to-end against real
  OpenRouter DeepSeek in ~76s for ~$0.05 per run. `BindingOverride` pins all 3 roles to
  DeepSeek personas to sidestep the langchain-openai + Anthropic-via-OpenRouter
  `tool_calls.args` JSON-string ValidationError (known v0.1.0 limit). New seed personas:
  `openrouter-deepseek-spec-writer@1` (capabilities: spec_write, phase_planning;
  max_cost_per_call_usd=0.01) and `openrouter-deepseek-code-reviewer@1` (capabilities:
  code_review, evidence_check; max_cost_per_call_usd=0.01). Persona count test updated
  to 12. `WorkflowEngine._build_envelope` now inlines the artifact JSON Schema directly
  in the prompt so the LLM sees exact required fields. `WorkflowEngine._record_llm_call`
  fills every NOT NULL `LlmCallRow` column (thread_id, persona_version, role, turn_index,
  cached_tokens, reasoning_tokens, cost_usd_input/output, etc.). `CostMiddleware` now
  probes both `usage_metadata` and `response_metadata.token_usage` (prompt_tokens /
  completion_tokens fallback) to capture OpenAI-compatible streamed responses forwarded
  by OpenRouter.
- Step 12 — Doctor full 8-check + OpenRouter pricing fetch: `mydeepagent doctor`
  now runs 8 checks (python / uv / git / workspace_root / config+governance /
  openrouter_api_key / openrouter_ping + pricing upsert / disk+sqlite integrity).
  `mydeepagent pricing` lists the cached OpenRouter pricing matrix from the
  persisted `model_pricing` table. `mydeepagent run` preview now reads from the
  persisted `model_pricing` table when populated, falling back to the static seed
  otherwise. 26 new tests (23 unit + 3 integration).
- Step 11 — Audit log + secret scrubbing: append-only `{state_dir}/audit.jsonl`
  recording every tool call (name/args/duration/error). `AuditToolMiddleware` now
  ships with a built-in JSONL recorder (`file_recorder`), attached automatically in
  `WorkflowEngine` and Interactive REPL. `structlog` configured project-wide via
  `my_deepagent.logging.configure_logging`, with a `_scrub_processor` that redacts
  OpenRouter / Anthropic / OpenAI / LangSmith / GitHub / GitLab API keys plus
  generic Bearer tokens before they reach stderr or JSON sinks. `audit.py` provides
  `append_audit_record` (O_APPEND, 0o600 permissions), `read_audit_records` (with
  optional limit, corrupt-line skip), and `make_audit_recorder` async factory.
  19 new tests (8 audit unit, 9 logging unit, 3 audit-middleware integration).
- Step 10 — Interactive REPL: `mydeepagent` (no subcommand) launches a prompt_toolkit
  REPL with `--agent` / `--model` overrides, slash commands (`/help`, `/quit`, `/exit`,
  `/agent`, `/model`, `/clear`, `/stats`, `/budget`, `/runs`), file refs
  (`@path/to/file.py` expansion with repo-root containment check), and
  `CostMiddleware`-wired agent calls so spending is metered per interactive session.
  `slash.py` implements `parse_slash` + `SlashRegistry`. `CostMiddleware` gains
  `interactive_session_id` parameter. 21 new tests (10 slash unit, 5 file-ref unit,
  3 CLI integration, 3 updated CLI unit).
- Step 9 — Crash recovery + concurrency: `sweep_orphan_runs(db)` in
  `my_deepagent.recovery` marks non-terminal runs/phases as failed at app startup so
  active-run uniqueness slots (partial unique index `ux_active_run_repo_base`) are freed;
  `mydeepagent runs list/show/resume` CLI in `my_deepagent.cli.runs` (list with optional
  `--state` filter, show by full UUID or 6+ char prefix, resume stub with exit-2 hint);
  SIGTERM/SIGINT graceful shutdown in `WorkflowEngine` (`install_signal_handlers`,
  `_on_signal`, `_force_cancel_inflight`; 30s grace then cancel in-flight tasks);
  auto-sweep on `mydeepagent run` before any new phase begins. 21 new tests.
- Step 8 — Budget guardrails: `BudgetTracker` (SQLite WAL ledger via `BudgetLedgerRow`,
  on_hit policy block/warn_continue/prompt, per-run + per-day + per-persona-daily
  scopes) in `my_deepagent.budget`; cost preview before `mydeepagent run` (rich table
  with per-phase est.) via `my_deepagent.monitoring.cost_estimator`;
  `CostMiddleware` integrated with `BudgetTracker` (pre-call assert + post-call record);
  `WorkflowEngine` accepts optional `budget_tracker` and `pricing` kwargs (backward-
  compatible); CLI: `mydeepagent budget` (ledger), `mydeepagent stats --by model|persona|day`,
  `mydeepagent costs` (alias); `--no-preview` flag on `mydeepagent run`.
  28 new tests.
- Step 7 — Workflow engine: `WorkflowEngine` in `my_deepagent.engine` orchestrates
  phase loop, artifact watcher (write_file/edit_file detection), jsonschema validation
  with one repair retry, approval gate, and final report compose (JSON + Markdown).
  `ArtifactWatcherMiddleware` in `my_deepagent.middleware.artifact_watcher` intercepts
  write_file/edit_file tool calls targeting the expected artifact path.
  `RunEventType` + `run_idempotency_key` in `my_deepagent.run_event` (closed event set,
  deterministic idempotency keys per plan v2.0 §13.1).
  `cli/run.py` exposes `mydeepagent run <workflow.yaml>`.
  `tui/approval.py` prompts the user for approve/reject/request_changes/abort.
  FK-safe persistence: WorkflowTemplateRow and AgentPersonaRow upserted before RunRow
  to satisfy SQLite FK ordering constraints.
  18 new tests: 12 engine unit/integration tests + 6 artifact watcher tests.
- Step 6 — Distribution: `mydeepagent init/login/logout/keys/doctor` CLI commands;
  platformdirs-based data dirs; OS keyring (macOS Keychain / Linux Secret Service /
  Windows Credential Store) for API keys via `my_deepagent.keys`; first-run
  governance consent in `governance.py`; secret resolution priority
  (config → env → keyring → error) in `my_deepagent.secrets`; i18n catalog
  (ko / en) under `my_deepagent.i18n` controlled by `MYDEEPAGENT_LANG`.
- persistence/models.py (P0-1): partial unique index `ux_active_run_repo_base` on `runs(repo_path, base_branch) WHERE state NOT IN ('completed','failed','aborted')` — prevents duplicate active runs per repo/branch
- persistence/models.py (P0-3): FK constraints added to `RunRow.template_id` (RESTRICT), `RunBindingRow.persona_id` (RESTRICT), `InteractiveSessionRow.persona_id` (RESTRICT), `RunEventRow.phase_id` (CASCADE), `ApprovalRequestRow.phase_id` (CASCADE), `ArtifactRow.phase_id` (CASCADE), `ToolCallRow.run_id/phase_id/interactive_session_id` (CASCADE), `LlmCallRow.run_id/phase_id/interactive_session_id` (CASCADE), `PhaseFeedbackRow.run_id/phase_id` (CASCADE)
- alembic/versions/839f2233e346: new migration adding partial unique index and all FK constraints above; uses SQLite table-rebuild pattern with PRAGMA foreign_keys=OFF/ON guard
- persistence/checkpointer.py (P0-4): removed `get_checkpointer` (leaking connection helper); only `get_checkpointer_ctx` context manager is now exported
- tests/integration/test_checkpointer.py: 5 tests for checkpointer ctx lifecycle (file creation, parent dir, connection cleanup, lock-free concurrent use)
- tests/integration/test_persistence.py: 7 new P0 verification tests (active-run partial index blocks/allows, cascade-delete of phase_feedback+run_phases, RESTRICT on template delete, index exists in sqlite_master)
- tests/unit/test_session.py: full rewrite to deepagents dataclass API — FilesystemPermission attribute access (.mode/.paths/.operations), build_backend type dispatch (5 cases), _map_operations deduplication (8 cases), _spec_to_permission mapping, updated _subagent_to_dict and _resolve_openrouter_api_key tests; 47 unit tests total
- tests/integration/test_openrouter_smoke.py: real OpenRouter/DeepSeek smoke test (3 tests, ~$0.001-$0.003/run, max_tokens=50); skipped automatically when no API key is configured; validates ChatOpenAI response, usage_metadata tokens, and deepagents CompiledStateGraph end-to-end
- pyproject.toml: registered `integration` pytest marker to silence --strict-markers error
- v0.1.0 scaffolding (Step 0): src/tests/docs trees, ruff/mypy/pre-commit/alembic config
- Seed assets copied to docs/schemas/ (personas/workflows/artifacts validated)
- Core module (Step 1): config, enums, errors, hash + unit tests
- Persona / Workflow / Binding module (Step 2): pydantic schemas, YAML loaders, deterministic auto-select, override, consent store with atomic write
- Step 1 review patches (P0/P1): exception chain context suppression, classmethod LSP fix, workspace_root realpath canonicalization, config_invalid error mapping

### Changed
- deepagents 0.6.1 LocalShellBackend + permissions conflict workaround: removed `permissions` block from all 10 seed personas; `SafetyShellMiddleware` now enforces destructive-command + secret-path policy at the tool layer for local_shell backend agents.
- `build_agent` automatically prepends `SafetyShellMiddleware` to every agent and skips `permissions` kwarg when `deepagents_backend == "local_shell"`.
- `SafetyShellMiddleware` extended with secret-path enforcement: `read_file`/`write_file`/`edit_file`/`ls` tool calls are blocked when `file_path`/`path` matches any `DENY_PATH_PATTERNS` glob (wcmatch GLOBSTAR|IGNORECASE|DOTGLOB).
- All env vars require `MYDEEPAGENT_` prefix (e.g. `MYDEEPAGENT_OPENROUTER_API_KEY`, `MYDEEPAGENT_BUDGET_DAILY_USD`). `.env.example` updated accordingly. This isolates my-deepagent's env namespace from other tools.
- Persona / Workflow / FilesystemPermission models now store list-valued fields as tuples (deep immutability — prevents post-construction mutation that would invalidate compute_hash()).

### Known limitations (v0.1.0)
- `usage_metadata` is sometimes empty for responses forwarded by OpenRouter (deepagents
  wraps the underlying ChatOpenAI response so token counts may not surface). The
  `CostMiddleware` recorder still fires and a `LlmCallRow` row is persisted, but
  `input_tokens` / `output_tokens` may read as 0 — the E2E test treats this as a known
  limit. v0.2 will probe more response shapes (raw chunks / callbacks).
- Anthropic models via OpenRouter currently fail with a `tool_calls.args` JSON-string
  vs dict ValidationError inside `langchain-openai`. Workaround: pin DeepSeek personas
  via `BindingOverride`. Tracking for v0.2.
- `mydeepagent runs resume <run_id>` is a stub (exit-2 hint only); workflow replay
  from a half-run state is not yet implemented.