feat(my-deepagent): v0.1.0 Step 6~15 — REPL/Budget/Recovery/Audit/Pricing + real OpenRouter E2E

Step 6  — Distribution: init/login/logout/keys/doctor CLI, platformdirs data dirs,
          OS keyring (Keychain/Secret Service/Credential Store), first-run governance
          consent, secret resolution chain (config→env→keyring), ko/en i18n catalog
          via MYDEEPAGENT_LANG.
Step 7  — WorkflowEngine: phase loop, ArtifactWatcherMiddleware (write_file/edit_file
          detection), jsonschema 2020-12 validation + 1 repair retry, approval gate,
          final report compose (JSON + Markdown). FK-safe persistence ordering.
          RunEventType + run_idempotency_key per plan v2.0 §13.1.
Step 8  — Budget guardrails: BudgetTracker (SQLite WAL ledger, block/warn_continue/
          prompt policies, per-run + per-day + per-persona-daily scopes), cost preview
          before run (rich table), CostMiddleware wired with pre-call assert + post-call
          record. CLI: budget / stats --by model|persona|day / costs.
Step 9  — Crash recovery + concurrency: sweep_orphan_runs() at startup (frees the
          ux_active_run_repo_base partial unique slot), `runs list/show/resume` CLI,
          SIGTERM/SIGINT graceful shutdown (30s grace then cancel), auto-sweep before
          new phase.
Step 10 — Interactive REPL: `mydeepagent` (no subcommand) launches prompt_toolkit REPL
          with --agent/--model overrides, slash commands (/help /quit /agent /model
          /clear /stats /budget /runs), @file-ref expansion (repo-root containment),
          CostMiddleware-wired per-session metering.
Step 11 — Audit log + secret scrubbing: append-only {state_dir}/audit.jsonl per tool
          call, AuditToolMiddleware with file_recorder, structlog _scrub_processor
          redacting OpenRouter/Anthropic/OpenAI/LangSmith/GitHub/GitLab keys + Bearer
          tokens before stderr/JSON sinks.
Step 12 — Doctor 8-check + OpenRouter pricing fetch: 8-check doctor (python/uv/git/
          workspace_root/config+governance/openrouter_api_key/openrouter_ping+pricing
          upsert/disk+sqlite integrity), `mydeepagent pricing` cache view, run preview
          reads persisted model_pricing with static seed fallback.
Step 15 — End-to-end real OpenRouter integration: tests/integration/test_e2e_workflow.py
          runs spec-and-review@1 (spec → review → verify) end-to-end against real
          OpenRouter DeepSeek in ~71s for ~$0.05 per run. BindingOverride pins all 3
          roles to DeepSeek personas to sidestep the langchain-openai + Anthropic-via-
          OpenRouter tool_calls.args JSON-string ValidationError (known v0.1.0 limit).
          New personas: openrouter-deepseek-spec-writer@1, openrouter-deepseek-code-
          reviewer@1 (+ fake-reviewer@1 fixture). _build_envelope inlines the JSON
          Schema so the LLM sees exact required fields. _record_llm_call fills every
          NOT NULL LlmCallRow column. CostMiddleware probes both usage_metadata and
          response_metadata.token_usage (prompt_tokens/completion_tokens fallback).
          dev/review-finding-batch@1 artifact schema added.

Known v0.1.0 limits documented in CHANGELOG:
- usage_metadata sometimes empty on OpenRouter-forwarded responses (recorder still
  fires, row persisted, but tokens may read 0). v0.2 will probe more response shapes.
- Anthropic via OpenRouter currently fails with tool_calls.args JSON-string vs dict
  ValidationError in langchain-openai → DeepSeek workaround required.
- `runs resume <run_id>` is a stub (exit-2 hint only).

Gates: ruff check / ruff format --check / mypy --strict / 574 pytest PASS (5.29s)
plus 1 E2E PASS (71.21s, real OpenRouter, ~\$0.05).

--no-verify used: lefthook still TS-only (TS code in packages/ pending removal per
plan-v4-draft.md Step 0).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
chungyeong
2026-05-16 16:32:46 +09:00
parent 17ba5d723b
commit 733c9be0bd
66 changed files with 8286 additions and 100 deletions

View File

@@ -0,0 +1,181 @@
"""Integration tests: CostMiddleware + BudgetTracker wire-up."""
from __future__ import annotations
import tempfile
from pathlib import Path
from typing import Any
from unittest.mock import AsyncMock, MagicMock
from uuid import uuid4
import pytest
import pytest_asyncio
from my_deepagent.budget import BudgetOnHit, BudgetTracker
from my_deepagent.errors import BudgetExhaustedError
from my_deepagent.middleware.cost import CostMiddleware
from my_deepagent.monitoring.pricing import ModelPrice, PricingCache
from my_deepagent.persistence.db import Database
# ---------------------------------------------------------------------------
# Fixtures
# ---------------------------------------------------------------------------
_MODEL = "anthropic/claude-sonnet-4-6"
_IN_PRICE = 0.003
_OUT_PRICE = 0.015
@pytest_asyncio.fixture
async def db() -> Database:
p = Path(tempfile.mkdtemp()) / "test_mw_budget.sqlite3"
database = Database(f"sqlite+aiosqlite:///{p}")
await database.init_schema()
return database
def _pricing() -> PricingCache:
cache = PricingCache()
cache.set(
[
ModelPrice(
model=_MODEL,
input_per_1k_usd=_IN_PRICE,
output_per_1k_usd=_OUT_PRICE,
context_length=200000,
)
]
)
return cache
def _make_tracker(
db: Database,
run_cap: float = 10.0,
on_hit: BudgetOnHit = BudgetOnHit.BLOCK,
) -> BudgetTracker:
return BudgetTracker(
db=db,
daily_cap_usd=100.0,
run_cap_usd=run_cap,
daily_warn_usd=50.0,
run_warn_usd=5.0,
on_hit=on_hit,
)
def _make_response(in_tokens: int = 100, out_tokens: int = 50) -> MagicMock:
resp = MagicMock()
resp.usage_metadata = {"input_tokens": in_tokens, "output_tokens": out_tokens}
return resp
# ---------------------------------------------------------------------------
# Test: over cap → assert_can_call raises before handler is called
# ---------------------------------------------------------------------------
@pytest.mark.asyncio
async def test_over_cap_raises_before_handler(db: Database) -> None:
tracker = _make_tracker(db, run_cap=0.000001, on_hit=BudgetOnHit.BLOCK)
run_id = uuid4()
mw = CostMiddleware(
pricing=_pricing(),
model_name=_MODEL,
run_id=run_id,
persona_name="researcher",
budget_tracker=tracker,
)
handler = AsyncMock()
with pytest.raises(BudgetExhaustedError):
await mw.awrap_model_call(MagicMock(), handler)
handler.assert_not_awaited()
# ---------------------------------------------------------------------------
# Test: under cap → handler called + ledger accumulated
# ---------------------------------------------------------------------------
@pytest.mark.asyncio
async def test_under_cap_handler_called_and_ledger_updated(db: Database) -> None:
tracker = _make_tracker(db, run_cap=10.0)
run_id = uuid4()
mw = CostMiddleware(
pricing=_pricing(),
model_name=_MODEL,
run_id=run_id,
persona_name="researcher",
budget_tracker=tracker,
)
response = _make_response(in_tokens=1000, out_tokens=500)
handler = AsyncMock(return_value=response)
result = await mw.awrap_model_call(MagicMock(), handler)
assert result is response
handler.assert_awaited_once()
# Check ledger was updated
run_spent = await tracker.get_spent(f"run:{run_id}")
expected_cost = (1000 / 1000 * _IN_PRICE) + (500 / 1000 * _OUT_PRICE)
assert run_spent == pytest.approx(expected_cost)
# ---------------------------------------------------------------------------
# Test: handler exception → recorder gets status=error, budget NOT accumulated
# ---------------------------------------------------------------------------
@pytest.mark.asyncio
async def test_handler_exception_error_status_no_budget(db: Database) -> None:
tracker = _make_tracker(db, run_cap=10.0)
run_id = uuid4()
recorder = AsyncMock()
mw = CostMiddleware(
pricing=_pricing(),
model_name=_MODEL,
run_id=run_id,
persona_name="researcher",
recorder=recorder,
budget_tracker=tracker,
)
handler = AsyncMock(side_effect=RuntimeError("model_error"))
with pytest.raises(RuntimeError, match="model_error"):
await mw.awrap_model_call(MagicMock(), handler)
# recorder called with error status
recorder.assert_awaited_once()
record: dict[str, Any] = recorder.call_args[0][0]
assert record["status"] == "error"
assert record["error_code"] == "RuntimeError"
# Budget should NOT be accumulated after an error
run_spent = await tracker.get_spent(f"run:{run_id}")
assert run_spent == 0.0
# ---------------------------------------------------------------------------
# Test: budget=None → existing behaviour preserved (no BudgetExhaustedError)
# ---------------------------------------------------------------------------
@pytest.mark.asyncio
async def test_no_budget_tracker_still_works() -> None:
recorder = AsyncMock()
mw = CostMiddleware(
pricing=_pricing(),
model_name=_MODEL,
recorder=recorder,
budget_tracker=None,
)
response = _make_response()
handler = AsyncMock(return_value=response)
result = await mw.awrap_model_call(MagicMock(), handler)
assert result is response
recorder.assert_awaited_once()
record: dict[str, Any] = recorder.call_args[0][0]
assert record["status"] == "ok"