feat: implement full-platform STT API (v2.3 consensus plan)
Batch+realtime transcription API: faster-whisper engine w/ EngineOwner (single GPU owner, OOM downgrade chain, persisted attempted_profiles), hardware-adaptive device manager (T0-T3 VRAM tiers), Redis/in-proc job queue w/ leases + crash recovery, postprocess (glossary/rules/LLM w/ egress guard), privacy-first result store (UUID keys, source deleted after transcribe), retention sweeper, API-key auth (HMAC digests, scopes, job ownership), WebSocket realtime lane (LocalAgreement), CLI (detect/transcribe/bench/serve/key), Docker, benchmark runner. 127 mock-based tests pass; ruff clean. Includes verification checklist and autoplan review notes.
This commit is contained in:
@@ -1,3 +1,4 @@
|
||||
<!-- /autoplan restore point: /root/.gstack/projects/lukehemmin-luke_scribe/feat-full-platform-restore.md -->
|
||||
# Consensus Implementation Plan: luke_scribe — 로컬 STT 전사 API
|
||||
|
||||
- **Status:** `pending approval` (consensus **v2.3** — v2.2 + §11 Open Q 후속확정·모호도 ~5% 재산출(2026-06-07); v2.1 합의 + CCG 외부리뷰(Codex/Gemini) 반영; §3.6 능력등급·§3.10 프로비저닝/WS/공유스토어/Colab)
|
||||
@@ -272,3 +273,85 @@ luke_scribe/
|
||||
|
||||
---
|
||||
*Consensus v2.3 — `pending approval`. 실행(team/ralph/autopilot)은 사용자의 별도 명시 승인이 있어야만 진행됩니다. 승인 전 소스 수정·커밋·실행 스킬 호출 없음.*
|
||||
---
|
||||
|
||||
## GSTACK REVIEW REPORT — /autoplan (2026-08-12)
|
||||
|
||||
> Auto-review pipeline: CEO → (Design: skipped, no UI scope) → Eng → DX. Dual voices: Codex (gpt-5.6-sol) + independent Claude reviewer. All intermediate decisions auto-decided by the 6 principles; taste decisions and user challenges surfaced below.
|
||||
|
||||
### CEO DUAL VOICES — CONSENSUS TABLE
|
||||
```
|
||||
Dimension Claude Codex Consensus
|
||||
──────────────────────────────────── ─────── ─────── ─────────
|
||||
1. Premises valid? part part DISAGREE (no workload evidence)
|
||||
2. Right problem to solve? yes yes CONFIRMED (KO+EN terminology quality)
|
||||
3. Scope calibration correct? — no DISAGREE (full platform = over-engineering)
|
||||
4. Alternatives sufficiently explored? part no DISAGREE (whisper.cpp/WhisperX bake-off missing)
|
||||
5. Competitive/market risks covered? part no DISAGREE (commodity wrappers)
|
||||
6. 6-month trajectory sound? part part DISAGREE (infra before validation)
|
||||
```
|
||||
**CEO findings (Codex, 12):** no workload evidence (critical); product framed around wrong value — terminology-quality system is the wedge (critical); P1–P5 over-engineering for unvalidated internal tool (critical); rebuilding existing inference products — bake-off needed (high); no defensible differentiation (high); hybrid gate not decision-grade (high); realtime=turbo is assumption not architecture (high); shared-GPU scheduling structurally unsound (critical); "1050 to H100" support is a liability (high); AC evade capacity decision (high); privacy-first is deletion choreography without threat model (high); feature sequencing backwards — glossary/rules/bench before realtime (high).
|
||||
|
||||
### ENG DUAL VOICES — CONSENSUS TABLE
|
||||
```
|
||||
Dimension Claude Codex Consensus
|
||||
──────────────────────────────────── ─────── ─────── ─────────
|
||||
1. Architecture sound? no no DISAGREE (shared-GPU lock, P0)
|
||||
2. Test coverage sufficient? no no DISAGREE (fault matrix, contention gate)
|
||||
3. Performance risks addressed? part no DISAGREE (VRAM≠latency)
|
||||
4. Security threats covered? part no DISAGREE (key hashing, traversal, SSRF)
|
||||
5. Error paths handled? no no DISAGREE (finally gaps, crash recovery)
|
||||
6. Deployment risk manageable? part part DISAGREE (job_timeout, lease/heartbeat)
|
||||
```
|
||||
**ENG findings (Codex 25 / Claude 10):** P0 blockers — GPU lock does not cross processes (one GPU-owning inference service required), device admission can oversubscribe, realtime capacity not derivable from VRAM, `finally` doesn't cover SIGKILL/OOM/orphaned queued cancellation, OOM chain not restart-safe (persist `attempted_profiles`), `job_timeout≥4h` insufficient (derive from duration×RTF or renewable leases), worker-crash reconstruction missing (lease + startup reconciler), LocalAgreement-2 not tight enough (token-span agreement, monotonic timestamps, immutable committed output). P1 — API-key hashing (HMAC digest, constant-time), result/upload path traversal (UUID keys, enum formats), WS init-frame resource exhaustion caps, egress SSRF via provider-IDs + no redirects + reject private IPs, cancellation must cover ingest/ffmpeg phases, empty-audio contract, queue full must reserve bytes too, ffmpeg streaming = subprocess protocol (backpressure, kill process group), VAD ownership per lane, postprocess must be span-aware to not invalidate timestamps.
|
||||
|
||||
### DX DUAL VOICES — CONSENSUS TABLE
|
||||
```
|
||||
Dimension Claude Codex Consensus
|
||||
──────────────────────────────────── ─────── ─────── ─────────
|
||||
1. Getting started < 5 min? — no DISAGREE (no README/sample/timed smoke gate)
|
||||
2. CLI naming guessable? — part CONFIRMED (detect/transcribe/bench/serve + doctor alias)
|
||||
3. Error messages actionable? — no DISAGREE (error contract with corrective commands)
|
||||
4. Docs findable & complete? — no DISAGREE (README/COLAB.md/samples/quickstart missing)
|
||||
5. Upgrade path safe? — n/a N/A
|
||||
6. Dev environment friction-free? — no DISAGREE (uv lockfile, .python-version, .env.example, key create)
|
||||
```
|
||||
**DX findings (Codex):** no timed clone→transcript gate (high); README+licensed sample audio must be P1 deliverables (high); `.python-version`+`uv.lock`+pip fallback (medium); `.env.example` unspecified keys (high); Colab notebook/COLAB.md early (high); CLI exit codes/help/stderr contract (medium); error messages need what/cause/fix+command (high); model provisioning UX (`models list/pull`, size/cache/progress) (high); API-key bootstrap `key create` print-once + digest storage (high); path traversal on result endpoint (critical); shared-GPU one-owning-process (critical); OpenAPI `/docs` + API-key scheme + curl flow (high); WS wire protocol versioned + clients (high); Docker cpu/gpu compose profiles + readiness + preflight matrix (high).
|
||||
|
||||
### CROSS-PHASE THEMES (2+ phases independently)
|
||||
- **Theme 1: Shared-GPU scheduling unsound** — flagged CEO(8), Eng(1), DX(critical). High-confidence signal. Fix: one GPU-owning inference service/process; API and workers submit work, never touch CUDA directly. For a mock-verified first build: batch-only lane through a single engine-owner abstraction.
|
||||
- **Theme 2: Scope = build before validate** — CEO(1,3,6), Eng(6), DX(gate). All three voices recommend a thin first release (batch CLI/API, terminology fixtures, one model, JSON/SRT, deletion) and deferring realtime/diarization/LLM/queue until usage proves need.
|
||||
- **Theme 3: Privacy/security contracts underspecified** — CEO(11), Eng(14–17), DX(key bootstrap). Fix: hashed keys, ownership on all job endpoints, UUID result keys, provider-ID egress, no tunnels/external LLM in default.
|
||||
|
||||
### USER CHALLENGE — Scope (never auto-decided)
|
||||
- **You said:** full platform (v2.3 consensus plan P1–P5) on a fresh branch.
|
||||
- **Both models recommend:** a thin first release — batch CLI + batch REST API, terminology benchmark fixtures, one model (turbo), JSON/SRT output, immediate source deletion, actionable errors, README/sample/5-min smoke gate. Defer realtime WS, Redis/RQ queue, diarization, external LLM, Colab tunnel until observed demand.
|
||||
- **Why:** 3 independent voices (Codex CEO/Eng/DX) converged; the plan itself gates hybrid on P1 bench; infra (queue, realtime, tiers) is speculative until terminology accuracy is measured; shared-GPU architecture needs a redesign (one GPU-owning process) that a thin release makes trivial.
|
||||
- **What we might be missing:** user has internal context — actual audio volume, who consumes results, whether realtime is a hard requirement, whether "complete platform" is a directive vs aspiration.
|
||||
- **If we're wrong, the cost is:** we build the thin wedge, and the user then spends another cycle adding queue/realtime. (Cost of going full-platform first and being wrong is higher: wasted scaffolding + architectural rework per Eng findings.)
|
||||
- **Exception note:** no security/feasibility blocker is claimed — this is a scope/preference challenge, not a correctness one. The user decides.
|
||||
|
||||
### AUTO-DECIDED (audit trail — 6 principles)
|
||||
| # | Phase | Decision | Classification | Principle | Rationale | Rejected |
|
||||
|---|-------|----------|-----------|-----------|----------|
|
||||
| 1 | CEO | Accept premise: internal STT tool, KO+EN terminology is the product value | Mechanical | P6 | Direction matches docs | — |
|
||||
| 2 | CEO | Keep hardware-adaptive design but no fixed HW targets | Mechanical | P1 | Already resolved in v2.3 §11 | — |
|
||||
| 3 | CEO | Reject multi-engine plugin (faster-whisper single) | Mechanical | P4/P5 | Duplicates nothing; thin interface | Plugin system |
|
||||
| 4 | CEO | Defer Colab tunnel, diarization, external LLM to later phases | Taste | P3 | All voices: defer until proven | Ship now |
|
||||
| 5 | Eng | One GPU-owning engine-owner abstraction for batch lane | Mechanical | P5 | Required to fix P0 scheduling flaw | Independent locks |
|
||||
| 6 | Eng | Persist `attempted_profiles`; cap OOM retries at 2 total | Mechanical | P1 | Restart-safe downgrade chain | Unlimited retry |
|
||||
| 7 | Eng | Job leases + startup reconciler for stale `processing` | Mechanical | P1 | Worker-crash recovery | Terminal-only sweeper |
|
||||
| 8 | Eng | API keys: hashed storage + constant-time compare + ownership | Mechanical | P1 | Security boundary §3.8 | Plaintext config |
|
||||
| 9 | Eng | UUID result keys + enum formats (no traversal) | Mechanical | P1 | Containment | Client filenames |
|
||||
| 10 | Eng | Cancellation covers ingest/ffmpeg + generator close | Mechanical | P1 | Cooperative cancel completeness | Segment-only |
|
||||
| 11 | DX | `luke-scribe key create` print-once + digest store | Mechanical | P1 | Bootstrap friction | Manual config |
|
||||
| 12 | DX | README + licensed sample audio + timed smoke gate as P1 deliverable | Taste | P1/P2 | 3 voices converge | Defer docs |
|
||||
|
||||
### NOT in scope (auto-deferred → TODOS.md)
|
||||
Realtime WebSocket lane, Redis/RQ durable queue, diarization, external LLM correction, Colab tunnel automation, SRT/VTT (P4), word timestamps (P4), multi-worker auto-sizing, multi-GPU, 30-min memory soak, destructive fault matrix (hardware-gated), shared-GPU contention gate (hardware-gated).
|
||||
|
||||
### What already exists
|
||||
- `origin/feat/p1-core`: partial implementation (devices/engine/audio/cli/api/postprocess) — **not merged**; per user decision we rebuild cleanly on `feat/full-platform` from `main`. Reference-only.
|
||||
|
||||
### DREAM STATE DELTA
|
||||
Current: planning docs only. This plan → thin-first-release core with terminology benchmark as the product wedge → 12-month ideal: validated terminology-quality system (corpus, glossary, correction feedback, eval automation) with swappable inference backend.
|
||||
|
||||
Reference in New Issue
Block a user