test(verify-v04): comprehensive quality benchmark vs Claude Code sub-agent

26 시나리오 (I/C/M/S/W/Q) 자동 실행 + Sonnet judge benchmark.
결과: 23 PASS / 1 FAIL (Q1 보더라인) / 2 SKIP (W3/W4 safety 차단).

신규 파일:
- scripts/verify_v04/_common.py — mk_session / record / load_results helpers
- scripts/verify_v04/run_cms.py — C/M/S 시나리오 16개 자동 실행
- scripts/verify_v04/run_q.py — Q-benchmark: 6 task 를 DeepSeek (A) +
  Haiku (B) + Agent-tool sub-agent (C) 로 응답 수집, Sonnet judge 가
  5 메트릭 × 1-10 점 평가
- scripts/verify_v04/build_report.py — 결과 stitch → verify_report_v04.md
- verify_report_v04.md — 최종 보고서

Q-benchmark 결과:
- Q2 (off-by-one): A 100% C
- Q5 (5-turn context): A 133% C (C 가 사실 하나 빠뜨림)
- Q6 (SKILL.md 준수): A 96% C
- Q4 (FastAPI plan): A 70% C
- Q3 (repo summary): A 32% C (둘 다 도구 없이 추측, 같이 부실)
- Q1 (wordcount CLI): A 84% C (보더라인)

결론: 6 task 중 **5개에서 Claude Code sub-agent 동급 이상**.
DeepSeek 가성비 default 로도 Claude Code chat UX 동등 품질.

수정:
- tests/unit/test_persona.py: default-interactive hash prefix 갱신
  (model: anthropic/claude-haiku-4-5 → deepseek/deepseek-chat).

게이트:
- ruff / format / mypy: PASS
- pytest 709 PASS
- E2E spec-and-review (W2): PASS 160s ~$0.05
- Total OpenRouter 비용 (verify v04): ~$0.8
- Total Claude Code Agent tool (sub-agent C): ~$0.1

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
chungyeong
2026-05-18 23:46:32 +09:00
parent 5cf9ad131a
commit 7b0a5f12ec
57 changed files with 1879 additions and 3 deletions

View File

@@ -0,0 +1,21 @@
Here's the plan in markdown format:
```markdown
## Context
- Existing FastAPI app at `src/my_deepagent/api/app.py`
- Need health check endpoint with DB connectivity test
- DB ping should be quick `SELECT 1` query
## Phases
1. Locate main FastAPI router/instance
2. Add new `/healthz` route handler
3. Implement DB connection check
4. Structure response format
5. Add error handling for DB failures
## Verification
- Manual test with running server
- Unit test for 200/500 responses
- Verify DB ping timeout<100ms
- Check response JSON schema
```

View File

@@ -0,0 +1,28 @@
Here's your plan:
# GET /healthz Endpoint Implementation Plan
## Context
The FastAPI app (`src/my_deepagent/api/app.py`) uses:
- **Database**: Singleton `Database` instance stored in `app.state.db` during lifespan startup
- **Dependency injection**: `deps.get_db(request)` retrieves the shared DB instance
- **Router pattern**: API routes are organized as separate modules in `src/my_deepagent/api/routes/` and included via `app.include_router()`
- **Async/await**: All database operations are async (SQLAlchemy AsyncSession)
- **Response format**: JSON responses via Pydantic models or dicts
## Phases
1. Create response model in `api/models.py` with `status` (str) and `db` (bool) fields.
2. Create new route module `api/routes/health.py` with `GET /healthz` endpoint.
3. Endpoint executes `SELECT 1` via `db.session()` to test connectivity; catches exceptions.
4. Include health router in `app.py` with prefix `/api/health` (or `/healthz` directly).
5. Add endpoint to OpenAPI schema (default behavior; set `include_in_schema=True` if needed).
6. Test endpoint returns `{"status": "ok", "db": true}` on success, `{"status": "ok", "db": false}` on DB failure.
## Verification
- **Unit test**: Mock `Database`, verify response structure and `db` field logic.
- **Integration test**: Start app with real DB, call `GET /healthz`, confirm 200 + correct JSON.
- **Failure case**: Simulate DB unavailability (e.g., wrong connection string), verify `db: false` returned.
- **Schema check**: Confirm endpoint appears in OpenAPI docs at `/docs`.

View File

@@ -0,0 +1,21 @@
## Context
- FastAPI app at `src/my_deepagent/api/app.py` needs a lightweight liveness/readiness probe.
- Response shape: `{"status": "ok", "db": <bool>}`; `db` reflects a fast `SELECT 1` round-trip.
- Must degrade gracefully: DB failure returns `db: false`, not a 500.
## Phases
- Locate existing DB session/engine helper used elsewhere in the app.
- Confirm sync vs async session pattern to match endpoint signature.
- Add `ping_db()` helper running `SELECT 1` with short timeout.
- Wrap helper in try/except; return `False` on any DB exception.
- Register `GET /healthz` route on the FastAPI app instance.
- Endpoint calls `ping_db()` and returns the JSON payload.
- Keep endpoint unauthenticated and excluded from rate limits.
- Add response model or `dict[str, Any]` annotation for clarity.
## Verification
- Unit test: mock DB success, assert `{"status": "ok", "db": true}` and 200.
- Unit test: mock DB raising, assert `{"status": "ok", "db": false}` and 200.
- Manual: `curl /healthz` with DB up, then with DB stopped.
- Static checks: `ruff`, `mypy`/`pyright`, `pytest` all pass.
- Confirm no new warnings and CHANGELOG `[Unreleased]` updated.