0630142c345e3f99f0b054270820d990605cbaa5
6 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
chungyeong
|
0630142c34 |
feat(my-deepagent): v0.2 PR #3 — FastAPI + SSE + minimal Web GUI (mydeepagent serve)
Closes the "GUI 미존재" gap from the user's first-session requirements (REPL + workflow + GUI). v0.2 PR #1's Postgres migration made a second concurrent writer safe; v0.2 PR #2a/#2b wired durable resume; this commit ships the HTTP + browser surface that uses them. No auth, no multi-tenant, single uvicorn worker — per DR-3 boundaries. v0.3+ will add auth, multi-worker fanout, LISTEN/NOTIFY SSE upgrade. Backend - `src/my_deepagent/api/`: - `app.py` create_app() factory. lifespan stores db/config/personas/ workflows on app.state. CORS allow_origin_regex http://localhost(:port)?. /static mount + /, /{page}.html for the HTML frontend. - `models.py` — pydantic v2 DTOs (extra="forbid") for every route. Auto OpenAPI/Swagger via FastAPI's response_model. - `deps.py` — get_db / get_config / get_personas / get_workflows. - `runner.py` — start_new_run / start_resume. Pre-allocates run_id via new `WorkflowEngine.run(pre_allocated_run_id=...)` so the route returns the id immediately while the engine runs in asyncio.create_task. - `sse.py` — 0.5 s poll over run_events.seq. Emits ServerSentEvent rows; sends `event: done` and HTTP-200-closes when run hits terminal. - `routes/{runs,personas,workflows,budget}.py`: GET /api/runs (list, ?limit + ?state) GET /api/runs/{id} (detail + phases + artifacts + events) POST /api/runs (start; mock-able via runner.start_new_run) POST /api/runs/{id}/resume POST /api/runs/{id}/abort GET /api/runs/{id}/events (SSE; Last-Event-ID header + ?last_event_id) GET /api/personas GET /api/workflows GET /api/budget CLI - `cli/serve.py` mydeepagent serve [--host 127.0.0.1] [--port 8000]. Loud stderr warning if --host is not loopback (no auth = footgun). uvicorn.run(factory=True, workers=1). - `cli/main.py` serve command registered. Static frontend (vanilla HTML/JS/CSS, no build system) - index.html — runs list + budget summary - new.html — start-run form (workflow select, repo path, requirements, per-role persona override) - run.html — run detail + live SSE event log + Resume/Abort buttons - app.js — fetch + EventSource. XSS policy HARDCODED at file top: textContent only, innerHTML/insertAdjacentHTML/outerHTML forbidden. - style.css — dark theme, single file. Engine - WorkflowEngine.run(... pre_allocated_run_id: UUID|None = None). None → uuid4() (existing behavior). Set → use that UUID. Backward compatible. Tests - tests/integration/test_api_read.py (5): list empty, get 404, personas seed count (12), workflows seed (>=3), budget empty. - tests/integration/test_api_write.py (5): missing template 400, extra field 422, resume 404, abort 404, mock-runner happy path. - tests/integration/test_api_sse.py (1): seed terminal run + 3 events, drain stream, assert types present + stream closes within 3 s. - tests/integration/test_api_static.py (5): index/new/run HTML 200, app.js content-type + XSS-policy substring assertion, style.css content-type. - All fixtures use httpx ASGITransport + app.router.lifespan_context (httpx does NOT auto-trigger FastAPI lifespan) + sqlite tmp_path. Gates - ruff check + ruff format --check + mypy --strict: PASS (120 source files) - pytest non-E2E: 603 PASS (12.15 s) — +16 from new API tests - pytest E2E real OpenRouter on Postgres: PASS 60.44 s (baseline 71–122 s range; well within DR-3 acceptance threshold ≤+20%) Manual browser verification deferred to a follow-up (docker compose up, mydeepagent serve, open http://localhost:8000). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
chungyeong
|
501292a5cd |
feat(my-deepagent): v0.2 PR #2b — mydeepagent runs resume <id> real implementation
Closes the v0.1.0 KNOWN LIMIT where resume was an exit-2 stub. Builds on
v0.2 PR #2a's LangGraph wiring + the existing DB phase-state machine +
sweep_orphan_runs — no Temporal (per DR-3).
Highlights
- `WorkflowEngine.resume(run_id)` (new async method):
- Loads RunRow, rejects terminal states with
MyDeepAgentError("run_already_terminal").
- Reloads worktree_root from `RunRow.worktree_root`, template via
`_reload_template` (WorkflowTemplateRow JOIN + model_validate), and
bindings via `_reload_bindings` (run_bindings ⨝ agent_personas).
- **Does NOT call `bind_personas` again** — locks in the original
binding so consent / persona-pool changes since the original run
don't silently shift role assignment.
- `_execute_run` (extracted shared phase loop): `run()` and `resume()`
both dispatch through it. Skips already-completed phases (emits
`phase.skipped` event) and re-executes the rest.
- 4 new private helpers on WorkflowEngine: `_get_run_or_raise`,
`_reload_template`, `_reload_bindings`, `_get_completed_phase_keys`.
- `RunEventType.RUN_RESUMED` and `PHASE_SKIPPED` are now actually
emitted (the enum members existed already).
- `cli/runs.py _runs_resume_async`: stub → real impl. Validates the run
exists + non-terminal, loads seed personas + artifact schemas from
`docs/schemas/`, constructs WorkflowEngine with an
"abort-on-new-approval" callback (resume should not silently re-prompt
the user — original gates already passed; a new gate means the
workflow has changed). Calls engine.resume(UUID(id)), prints final
state + report. Catches MyDeepAgentError and exits 1 with red error.
Tests
- `tests/integration/test_resume.py` (new, 5 scenarios):
1. 2-phase mock workflow: phase 1 succeeds, phase 2 fails first time,
row flipped back to executing → resume → phase 2 completes.
Asserts `phase.skipped` event for phase 1, `run.resumed` event,
and exactly 1 mock invocation for phase 2 on resume.
2. Terminal run → `MyDeepAgentError(code="run_already_terminal")`.
3. Unknown run id → `MyDeepAgentError(code="run_not_found")`.
4. RunBindingRow rows missing → `MyDeepAgentError(code="run_metadata_missing")`.
5. Corrupt `workflow_templates.definition` →
`MyDeepAgentError(code="template_load_failed")`.
Mock pattern matches existing test_engine.py: patch
`my_deepagent.engine.build_agent` to return a fake agent that writes
the expected artifact and drives the watcher middleware.
Gates
- ruff check + ruff format --check + mypy --strict: PASS (103 source files)
- pytest non-E2E: 587 PASS (12.69 s) — +5 from new resume tests
- pytest E2E real OpenRouter on Postgres: PASS 78.52 s (baseline 71–122 s;
within DR-3 acceptance threshold ≤+20%)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
chungyeong
|
50aacd3382 |
feat(my-deepagent): v0.2 PR #2a — wire LangGraph AsyncPostgresSaver into engine
Foundation for `runs resume` (v0.2 PR #2b). v0.2 PR #1 added langgraph-checkpoint-postgres as a dependency, but engine.py did not yet pass `checkpointer=` to `build_agent` or set the LangGraph `thread_id` in `agent.ainvoke` — meaning resume had no state to restore. This commit actually wires the dependency. Highlights - `WorkflowEngine.__init__` accepts `checkpointer_url: str | None` (default = `config.database_url`). - `_maybe_open_saver` async context: opens AsyncPostgresSaver for postgresql{,+asyncpg,+psycopg}:// URLs; yields None for `sqlite+aiosqlite://` (test affordance — production always Postgres per DR-2 / DR-3, no langgraph-checkpoint-sqlite in deps). - `WorkflowEngine.run()` opens the saver **once per run** and shares it across all phases. Opening per-phase would reconnect 5+ times for no isolation gain — LangGraph checkpoints are keyed by `thread_id`, not by saver instance. - `_invoke_agent_until_artifact` forwards `checkpointer=self._saver` to `build_agent` and passes `config={"configurable": {"thread_id": f"run:<uuid>:phase:<uuid>"}}` to `agent.ainvoke`. The thread_id format is already used by `LlmCallRow.thread_id` (cost ledger), so a single key namespace covers both cost tracking and checkpoint replay. Tests - `tests/integration/test_engine_checkpointer_wiring.py` (new, 2 tests): 1. Engine wiring contract: spy `build_agent` to capture kwargs, assert `checkpointer` is non-None and `agent.ainvoke` receives the expected `config.configurable.thread_id` in run:<uuid>:phase:<uuid> format. 2. LangGraph thread isolation: distinct thread_ids write to independent rows in the auto-created `checkpoints` table; aput / aget round-trip preserves per-thread identity (sanity check against future deepagents wrap regressions). - `tests/integration/test_engine.py`: 5 mock-agent tests had fake `_ainvoke(messages)` signatures; widened to `(messages, **_kwargs)` to accept the new `config=` arg without behavior change. Gates - ruff check + ruff format --check + mypy --strict: PASS (103 source files) - pytest non-E2E: 582 PASS (10.55 s) — was 576 before, +7 from new wiring tests, +/-1 from engine.py reshape, +/-... settled at 582 net. - pytest E2E real OpenRouter on Postgres: PASS 75.99 s (baseline 71–122 s; within DR-3 acceptance threshold ≤+20%). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
chungyeong
|
e21a5241bf |
feat(my-deepagent): v0.2 PR #1 — Postgres migration (ahead of M8-Py FastAPI)
Switches the production backing store from SQLite to PostgreSQL 16, per DR-2.
The migration trigger is two concurrent writers on the my-deepagent ORM
tables — which first appears with FastAPI (M8-Py). Doing the cut now keeps
the surface area small while M8-Py is still planning.
Production deps: `asyncpg`, `psycopg[binary]`, `langgraph-checkpoint-postgres`.
Test deps: `aiosqlite` (the bulk of unit + integration tests stay on sqlite
tmp_path for speed; the E2E suite and the new checkpointer tests exercise
the live Postgres path).
Highlights
- `persistence/db.py`: dialect-aware connect listener. SQLite still gets
WAL + busy_timeout=5000 + foreign_keys=ON; Postgres gets `SET TIME ZONE 'UTC'`.
Added `Database.dialect_name` + `drop_schema` (test-only).
- `persistence/checkpointer.py`: SqliteSaver → AsyncPostgresSaver. API is
now async (`async with`) and takes a connection string. SQLAlchemy URL
prefixes (`+asyncpg`, `+psycopg`) are auto-stripped to a plain libpq DSN
(`_to_psycopg_dsn` helper, 4 unit tests).
- `persistence/upsert.py` (new): `insert_for(session)` — dialect-aware UPSERT
helper. Picks `postgresql.insert` or `sqlite.insert` based on the bound
engine. Replaces 5 hardcoded `sqlite_insert` call sites in `budget.py`,
`recovery.py`, `cli/doctor.py`.
- `persistence/models.py`: `RunRow` partial unique index declares both
`postgresql_where=` and `sqlite_where=` for cross-dialect correctness.
- `config.py`: default `database_url` now
`postgresql+asyncpg://devflow:devflow@localhost:55432/mydeepagent`. v3
`devflow` DB preserved untouched; v4 lives in a fresh `mydeepagent` DB.
- `cli/doctor.py` check 8: dialect-aware DB liveness probe. Postgres path
runs `SELECT 1` (pg_isready equivalent); SQLite keeps `PRAGMA integrity_check`.
- `alembic/env.py`: env-aware URL resolution (`MYDEEPAGENT_DATABASE_URL` >
`DATABASE_URL` > default). Async driver prefixes are mapped to the sync
equivalents alembic needs.
- `alembic/versions/9f2a6c79667e_v0_2_baseline_schema_postgres.py` (new):
fresh baseline autogenerated against live Postgres. Old SQLite migrations
(`79945fdc2649`, `839f2233e346`) deleted — v0.2 starts a clean history.
- `tests/conftest.py` (new): `pg_db_url` async fixture creates a fresh DB
per test against docker-compose `devflow-postgres` and drops it on
teardown after terminating lingering backends.
- `tests/integration/test_checkpointer.py`: rewritten for AsyncPostgresSaver
(4 pure DSN-converter unit tests + 3 async context-manager integration tests).
- `tests/integration/test_e2e_workflow.py`: switched to `pg_db_url`. Real
OpenRouter E2E now exercises the production Postgres path end-to-end.
Recovery
- Previous SQLite database at the platformdirs data_dir is NOT auto-migrated;
v0.1.0 was the only release that wrote to it. Set
`MYDEEPAGENT_DATABASE_URL=sqlite+aiosqlite:///<path>` to read it.
- The v3 `devflow` Postgres DB is preserved untouched (separate database
name); to inspect: `psql -h localhost -p 55432 -U devflow -d devflow`.
Gates
- ruff check + ruff format --check + mypy --strict: PASS (102 source files)
- pytest non-E2E: 576 PASS (5.46 s)
- pytest E2E real OpenRouter on Postgres: 1 PASS (122.93 s, ~$0.05/run)
--no-verify: lefthook still TS-only (deleted in
|
||
|
chungyeong
|
733c9be0bd |
feat(my-deepagent): v0.1.0 Step 6~15 — REPL/Budget/Recovery/Audit/Pricing + real OpenRouter E2E
Step 6 — Distribution: init/login/logout/keys/doctor CLI, platformdirs data dirs,
OS keyring (Keychain/Secret Service/Credential Store), first-run governance
consent, secret resolution chain (config→env→keyring), ko/en i18n catalog
via MYDEEPAGENT_LANG.
Step 7 — WorkflowEngine: phase loop, ArtifactWatcherMiddleware (write_file/edit_file
detection), jsonschema 2020-12 validation + 1 repair retry, approval gate,
final report compose (JSON + Markdown). FK-safe persistence ordering.
RunEventType + run_idempotency_key per plan v2.0 §13.1.
Step 8 — Budget guardrails: BudgetTracker (SQLite WAL ledger, block/warn_continue/
prompt policies, per-run + per-day + per-persona-daily scopes), cost preview
before run (rich table), CostMiddleware wired with pre-call assert + post-call
record. CLI: budget / stats --by model|persona|day / costs.
Step 9 — Crash recovery + concurrency: sweep_orphan_runs() at startup (frees the
ux_active_run_repo_base partial unique slot), `runs list/show/resume` CLI,
SIGTERM/SIGINT graceful shutdown (30s grace then cancel), auto-sweep before
new phase.
Step 10 — Interactive REPL: `mydeepagent` (no subcommand) launches prompt_toolkit REPL
with --agent/--model overrides, slash commands (/help /quit /agent /model
/clear /stats /budget /runs), @file-ref expansion (repo-root containment),
CostMiddleware-wired per-session metering.
Step 11 — Audit log + secret scrubbing: append-only {state_dir}/audit.jsonl per tool
call, AuditToolMiddleware with file_recorder, structlog _scrub_processor
redacting OpenRouter/Anthropic/OpenAI/LangSmith/GitHub/GitLab keys + Bearer
tokens before stderr/JSON sinks.
Step 12 — Doctor 8-check + OpenRouter pricing fetch: 8-check doctor (python/uv/git/
workspace_root/config+governance/openrouter_api_key/openrouter_ping+pricing
upsert/disk+sqlite integrity), `mydeepagent pricing` cache view, run preview
reads persisted model_pricing with static seed fallback.
Step 15 — End-to-end real OpenRouter integration: tests/integration/test_e2e_workflow.py
runs spec-and-review@1 (spec → review → verify) end-to-end against real
OpenRouter DeepSeek in ~71s for ~$0.05 per run. BindingOverride pins all 3
roles to DeepSeek personas to sidestep the langchain-openai + Anthropic-via-
OpenRouter tool_calls.args JSON-string ValidationError (known v0.1.0 limit).
New personas: openrouter-deepseek-spec-writer@1, openrouter-deepseek-code-
reviewer@1 (+ fake-reviewer@1 fixture). _build_envelope inlines the JSON
Schema so the LLM sees exact required fields. _record_llm_call fills every
NOT NULL LlmCallRow column. CostMiddleware probes both usage_metadata and
response_metadata.token_usage (prompt_tokens/completion_tokens fallback).
dev/review-finding-batch@1 artifact schema added.
Known v0.1.0 limits documented in CHANGELOG:
- usage_metadata sometimes empty on OpenRouter-forwarded responses (recorder still
fires, row persisted, but tokens may read 0). v0.2 will probe more response shapes.
- Anthropic via OpenRouter currently fails with tool_calls.args JSON-string vs dict
ValidationError in langchain-openai → DeepSeek workaround required.
- `runs resume <run_id>` is a stub (exit-2 hint only).
Gates: ruff check / ruff format --check / mypy --strict / 574 pytest PASS (5.29s)
plus 1 E2E PASS (71.21s, real OpenRouter, ~\$0.05).
--no-verify used: lefthook still TS-only (TS code in packages/ pending removal per
plan-v4-draft.md Step 0).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
chungyeong
|
17ba5d723b |
feat(my-deepagent): v0.1.0 Step 0~5 — scaffolding through deepagent + OpenRouter
Python rewrite of the agent harness on top of deepagents 0.6.1 + langchain 1.x, replacing the abandoned TS attempt in packages/. 388 unit/integration tests pass. Steps ----- 0. Scaffolding — uv workspace, ruff/mypy/pre-commit/alembic, src/tests/docs trees with docs/schemas/ seeded from my-deepagent-seed/. 1. Core — config (pydantic-settings with MYDEEPAGENT_ env prefix and TOML source), enums (Backend, Capability, RiskLevel, ApprovalDecisionAction, ApprovalState, RunState, RunPhaseState, SessionState, ErrorClass), errors (MyDeepAgentError + BudgetExhaustedError with PEP-3134 cause + context suppression), hash (canonical JSON + sha256). 2. Persona/Workflow/Binding — pydantic v2 schemas with tuple-based deep immutability (post-construction hash drift prevented), YAML loaders, deterministic auto-select (preferred_backends → version → name → hash), override resolution with ineligibility diagnostics, PersonaConsentStore with fcntl.flock + tmp+fsync+rename atomic write. 3. Artifact schema registry — Draft202012Validator, multi-root resolution, structured ValidationFinding output. 4. Persistence — 18 SQLAlchemy 2.0 async ORM models with FK CASCADE/RESTRICT, WAL + busy_timeout + foreign_keys PRAGMA, alembic baseline + ux_active_run_repo_base partial unique index, LangGraph SqliteSaver as context manager only (lifecycle safety). 5. DeepAgent session — build_agent wires Persona → create_deep_agent with LocalShellBackend / FilesystemBackend / StateBackend / CompositeBackend, ChatOpenAI(base_url=openrouter) for openrouter: model strings, and 4 middleware classes (cost / audit-tool / safety-shell / fallback-model). Critical workarounds -------------------- - deepagents 0.6.1 rejects FilesystemPermission together with backends that implement SandboxBackendProtocol (LocalShellBackend). SafetyShellMiddleware enforces destructive-command and secret-path policy at the tool layer instead, and build_agent strips the permissions kwarg when the persona's deepagents_backend is local_shell. - FilesystemOperation in deepagents is Literal['read', 'write'] only; _map_operations collapses our richer schema (read/write/edit/ls) safely. Real OpenRouter smoke --------------------- test_openrouter_deepagents_local_shell_smoke calls DeepSeek via deepagents + LocalShellBackend + SafetyShellMiddleware end-to-end. PASS, ~$0.000001 cost, input=9 / output=1 tokens with content "OK". Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |