chungyeong/dev-puppeteer
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
32 Commits 2 Branches 0 Tags
f78b26dc6903baf54f7059b80fc3973f5702d09a
Commit Graph

4 Commits

Author SHA1 Message Date
chungyeong
f78b26dc69 feat(my-deepagent): v0.3 PR #2 — context compaction (auto + manual /compact) 
Claude Code의 auto-compact + `/compact` 슬래시 등가.

핵심 동작:
- 세션 누적 토큰 (`total_input_tokens + total_output_tokens`)이 활성 모델
  컨텍스트 윈도우의 70%를 넘으면 자동으로 가장 오래된 비-system / 비-archived
  메시지를 cheap 모델 (`openrouter:deepseek/deepseek-chat` 기본)로 1회 요약 →
  `MessageRow(is_summary=True, role=system)` 1줄 삽입 + 원본은 `archived=True`
  + negative seq band (-(original.seq + 1))으로 옮김.
- LangGraph thread는 `thread_suffix` bump로 새 컨텍스트 시작 (재인입 비용 회피).
  세션 자체는 살아있음 — `sessions show <id> --all`로 archived 메시지 조회 가능.
- 수동 `/compact` 슬래시도 동일 함수 호출.  메시지가 부족하면 (`< MIN_COMPACTABLE`)
  사유 출력하고 no-op.

데이터·라이브러리:
- `monitoring/token_budget.py` (신규): `tiktoken cl100k_base`로 추정 (DeepSeek/
  Anthropic 모델 정확한 토크나이저가 없으므로 보수적 over-count).
  `MODEL_CONTEXT_LIMITS` (DeepSeek 64k, Claude Sonnet/Haiku/Opus 200k, GPT-4o
  128k), 미등록 모델은 32k 기본값.  `COMPACTION_THRESHOLD = 0.7`.
- `compaction.py` (신규): `should_compact()` / `compact_session()` /
  `CompactionResult`.  `_SESSION_LOCKS: dict[str, asyncio.Lock]` 세션별 직렬화 —
  동시 compaction은 두 번째가 첫 번째를 기다림.  `KEEP_RECENT_K = 10`,
  `MIN_COMPACTABLE = 4`.  LLM 호출은 DB session 바깥 (asyncpg connection
  점유 회피).
- `pyproject.toml`: `tiktoken>=0.7` 명시 (이전엔 langchain-openai 경유 transitive).

REPL 통합 (`cli/interactive.py`):
- `_approx_token_count`를 tiktoken-based로 교체.
- 매 ainvoke 후 `should_compact(session_row)` → 임계 초과 시 자동
  `compact_session()` → 성공 시 `clear_agent_cache()`로 thread bump + 한 줄 알림.
- `/compact` 슬래시 등록 (`_register_compaction_slash`).

테스트 (`tests/integration/test_compaction.py`, 7 케이스):
1. `should_compact` 70% 임계 아래/위/미등록 모델 (3개)
2. `MIN_COMPACTABLE` 미만 → LLM 호출 없이 거부
3. Happy path: 14 메시지 → 4 archive(negative seq) + summary at seq=1 + 10 live
   유지 + 토큰 카운터 산술 검증
4. 동일 session_id 동시 호출 2개 → Lock 직렬화 검증
5. 없는 session_id → `session_not_found`

게이트:
- ruff check / format --check / mypy: PASS
- pytest -q --ignore=tests/integration/test_e2e_workflow.py
  --ignore=tests/integration/test_openrouter_smoke.py: 611 passed (7 신규 포함)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-17 20:28:11 +09:00
chungyeong
f8335e4515 feat(my-deepagent): v0.3 PR #1 — interactive session persistence + LangGraph saver wiring 
v0.3의 토대. REPL/GUI 둘 다 장기 대화를 영속해서 `mydeepagent --session <id>`
또는 `GET /api/sessions/{id}`로 어디서든 이어 진행 가능. Claude Code의
`claude --resume` 등가 능력.

Data model
- `persistence/models.py`:
  - 신규 `MessageRow` 테이블 — (session_id, seq) UNIQUE, role/content/
    tool_calls/token_count/is_summary/archived/ts. LangGraph checkpoint =
    source of truth, 이 테이블은 GUI/CLI 빠른 조회 mirror. divergence
    rebuild 매커니즘 없음 (단순성 우선).
  - `InteractiveSessionRow` 컬럼 8개 추가:
      total_input_tokens, total_output_tokens (PR #2 tiktoken으로 정밀화 예정),
      model, project_key (sha256(realpath(repo_path))[:16]),
      title (첫 user msg 50자), plan_mode (PR #5), parent_session_id (PR #6),
      depth (PR #6 sub-agent depth ≤ 3).
- `alembic/versions/684e70f4536a_*.py` (신규):
  - `op.batch_alter_table` 사용 — SQLite ALTER constraint 미지원 우회. Postgres는
    native DDL.
  - 자동생성이 제안한 LangGraph 테이블 (`checkpoints` 등) drop 라인은 의도적으로
    제거 (langgraph-checkpoint-postgres가 자체 관리).
  - server_default 박아서 기존 row 안전.

CLI
- `cli/interactive.py`:
  - REPL 진입 시 `get_checkpointer_ctx(config.database_url)` 컨텍스트 열고
    REPL 전체 동안 유지. `build_agent(..., checkpointer=saver)`로 deepagents에
    LangGraph saver wire. v0.2 PR #10의 CostMiddleware / AuditToolMiddleware
    보존.
  - `_invoke_and_stream`이 ainvoke 전후 명시적 MessageRow insert
    (user → ainvoke → assistant). last_message_at + total_*_tokens 누적 +
    첫 user msg로 title 자동 setter.
  - `InteractiveSession.thread_suffix` 도입. /model / /agent / /clear 호출
    시 suffix bump → LangGraph thread_id = `{session_id}:{suffix}` 로 새
    deepagents 컨텍스트 시작 (compaction과 같은 패턴, PR #2 재사용).
  - 신규 `--session <id|prefix>` 옵션: 기존 row 로드 (ended이면 거부) 또는
    신규 row insert (AgentPersonaRow upsert + project_key 박음).
  - `/clear` 슬래시 갱신: messages.archived=True + 새 thread 시작. 세션 자체
    는 살아있음 — `sessions show <id> --all`로 조회 가능.
- `cli/sessions.py` (신규): `mydeepagent sessions list/show/resume/end`.
  show <id> [--all]이 archived 메시지까지. 6+ char prefix + 중복 시 명시
  에러.
- `cli/main.py`: --session 옵션 + sessions 서브명령 + interactive_command
  시그니처 확장.

HTTP API
- `api/models.py`: SessionSummary / MessageInfo / SessionDetail /
  CreateSessionRequest / PostMessageRequest / SessionAck DTO 신규 (모두
  extra="forbid").
- `api/routes/sessions.py` (신규):
    GET  /api/sessions?limit=&state=
    GET  /api/sessions/{id}?all=true     (마지막 200 메시지)
    POST /api/sessions                    (persona_name, model_override, repo_path)
    POST /api/sessions/{id}/messages      (사용자 메시지 append, 동기 persist;
                                            PR #7 GUI에서 background ainvoke 추가)
    GET  /api/sessions/{id}/stream        (SSE — 0.5s polling, last-event-id 헤더
                                            + ?last_seq 둘 다 지원)
    POST /api/sessions/{id}/end
- `api/app.py`: sessions 라우터 마운트.

Tests
- `tests/integration/test_session_persist.py` (5 시나리오):
    1. create + post → row + 메시지 + title + token 누적 영속
    2. list가 신규 3 세션 모두 포함
    3. prefix resolution + 404
    4. end 후 메시지 거부 (409)
    5. ?all=true가 archived 메시지 surfacing

Gates
- ruff check + ruff format + mypy --strict: PASS (124 source files)
- pytest non-E2E: 608 PASS (25.86 s) — v0.2 PR #3 후 603에서 +5 신규
- pytest E2E real OpenRouter on Postgres: PASS 82.07 s (베이스라인 60–122s
  범위 내; DR-3 +20% 임계점 통과)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-17 20:06:21 +09:00
chungyeong
733c9be0bd feat(my-deepagent): v0.1.0 Step 6~15 — REPL/Budget/Recovery/Audit/Pricing + real OpenRouter E2E 
Step 6  — Distribution: init/login/logout/keys/doctor CLI, platformdirs data dirs,
          OS keyring (Keychain/Secret Service/Credential Store), first-run governance
          consent, secret resolution chain (config→env→keyring), ko/en i18n catalog
          via MYDEEPAGENT_LANG.
Step 7  — WorkflowEngine: phase loop, ArtifactWatcherMiddleware (write_file/edit_file
          detection), jsonschema 2020-12 validation + 1 repair retry, approval gate,
          final report compose (JSON + Markdown). FK-safe persistence ordering.
          RunEventType + run_idempotency_key per plan v2.0 §13.1.
Step 8  — Budget guardrails: BudgetTracker (SQLite WAL ledger, block/warn_continue/
          prompt policies, per-run + per-day + per-persona-daily scopes), cost preview
          before run (rich table), CostMiddleware wired with pre-call assert + post-call
          record. CLI: budget / stats --by model|persona|day / costs.
Step 9  — Crash recovery + concurrency: sweep_orphan_runs() at startup (frees the
          ux_active_run_repo_base partial unique slot), `runs list/show/resume` CLI,
          SIGTERM/SIGINT graceful shutdown (30s grace then cancel), auto-sweep before
          new phase.
Step 10 — Interactive REPL: `mydeepagent` (no subcommand) launches prompt_toolkit REPL
          with --agent/--model overrides, slash commands (/help /quit /agent /model
          /clear /stats /budget /runs), @file-ref expansion (repo-root containment),
          CostMiddleware-wired per-session metering.
Step 11 — Audit log + secret scrubbing: append-only {state_dir}/audit.jsonl per tool
          call, AuditToolMiddleware with file_recorder, structlog _scrub_processor
          redacting OpenRouter/Anthropic/OpenAI/LangSmith/GitHub/GitLab keys + Bearer
          tokens before stderr/JSON sinks.
Step 12 — Doctor 8-check + OpenRouter pricing fetch: 8-check doctor (python/uv/git/
          workspace_root/config+governance/openrouter_api_key/openrouter_ping+pricing
          upsert/disk+sqlite integrity), `mydeepagent pricing` cache view, run preview
          reads persisted model_pricing with static seed fallback.
Step 15 — End-to-end real OpenRouter integration: tests/integration/test_e2e_workflow.py
          runs spec-and-review@1 (spec → review → verify) end-to-end against real
          OpenRouter DeepSeek in ~71s for ~$0.05 per run. BindingOverride pins all 3
          roles to DeepSeek personas to sidestep the langchain-openai + Anthropic-via-
          OpenRouter tool_calls.args JSON-string ValidationError (known v0.1.0 limit).
          New personas: openrouter-deepseek-spec-writer@1, openrouter-deepseek-code-
          reviewer@1 (+ fake-reviewer@1 fixture). _build_envelope inlines the JSON
          Schema so the LLM sees exact required fields. _record_llm_call fills every
          NOT NULL LlmCallRow column. CostMiddleware probes both usage_metadata and
          response_metadata.token_usage (prompt_tokens/completion_tokens fallback).
          dev/review-finding-batch@1 artifact schema added.

Known v0.1.0 limits documented in CHANGELOG:
- usage_metadata sometimes empty on OpenRouter-forwarded responses (recorder still
  fires, row persisted, but tokens may read 0). v0.2 will probe more response shapes.
- Anthropic via OpenRouter currently fails with tool_calls.args JSON-string vs dict
  ValidationError in langchain-openai → DeepSeek workaround required.
- `runs resume <run_id>` is a stub (exit-2 hint only).

Gates: ruff check / ruff format --check / mypy --strict / 574 pytest PASS (5.29s)
plus 1 E2E PASS (71.21s, real OpenRouter, ~\$0.05).

--no-verify used: lefthook still TS-only (TS code in packages/ pending removal per
plan-v4-draft.md Step 0).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-16 16:32:46 +09:00
chungyeong
17ba5d723b feat(my-deepagent): v0.1.0 Step 0~5 — scaffolding through deepagent + OpenRouter 
Python rewrite of the agent harness on top of deepagents 0.6.1 + langchain 1.x,
replacing the abandoned TS attempt in packages/. 388 unit/integration tests pass.

Steps
-----
0. Scaffolding — uv workspace, ruff/mypy/pre-commit/alembic, src/tests/docs
   trees with docs/schemas/ seeded from my-deepagent-seed/.
1. Core — config (pydantic-settings with MYDEEPAGENT_ env prefix and TOML
   source), enums (Backend, Capability, RiskLevel, ApprovalDecisionAction,
   ApprovalState, RunState, RunPhaseState, SessionState, ErrorClass),
   errors (MyDeepAgentError + BudgetExhaustedError with PEP-3134 cause +
   context suppression), hash (canonical JSON + sha256).
2. Persona/Workflow/Binding — pydantic v2 schemas with tuple-based deep
   immutability (post-construction hash drift prevented), YAML loaders,
   deterministic auto-select (preferred_backends → version → name → hash),
   override resolution with ineligibility diagnostics, PersonaConsentStore
   with fcntl.flock + tmp+fsync+rename atomic write.
3. Artifact schema registry — Draft202012Validator, multi-root resolution,
   structured ValidationFinding output.
4. Persistence — 18 SQLAlchemy 2.0 async ORM models with FK CASCADE/RESTRICT,
   WAL + busy_timeout + foreign_keys PRAGMA, alembic baseline +
   ux_active_run_repo_base partial unique index, LangGraph SqliteSaver as
   context manager only (lifecycle safety).
5. DeepAgent session — build_agent wires Persona → create_deep_agent with
   LocalShellBackend / FilesystemBackend / StateBackend / CompositeBackend,
   ChatOpenAI(base_url=openrouter) for openrouter: model strings, and 4
   middleware classes (cost / audit-tool / safety-shell / fallback-model).

Critical workarounds
--------------------
- deepagents 0.6.1 rejects FilesystemPermission together with backends that
  implement SandboxBackendProtocol (LocalShellBackend). SafetyShellMiddleware
  enforces destructive-command and secret-path policy at the tool layer
  instead, and build_agent strips the permissions kwarg when the persona's
  deepagents_backend is local_shell.
- FilesystemOperation in deepagents is Literal['read', 'write'] only;
  _map_operations collapses our richer schema (read/write/edit/ls) safely.

Real OpenRouter smoke
---------------------
test_openrouter_deepagents_local_shell_smoke calls DeepSeek via deepagents +
LocalShellBackend + SafetyShellMiddleware end-to-end. PASS, ~$0.000001 cost,
input=9 / output=1 tokens with content "OK".

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-15 19:40:02 +09:00