Add outline import feature; drop Dto naming, map entities at the API boundary

Services now return entities; endpoints (and the agent toolsets) map to
*Response records instead of services building wire DTOs themselves.
Also brings in the outline-import agent, MCP tool, ledger and web dialog
that were already in progress on disk.
This commit is contained in:
James Wampler
2026-08-06 18:36:40 -07:00
parent 40f93e40a8
commit 189ebf3237
66 changed files with 3310 additions and 364 deletions
@@ -0,0 +1,47 @@
# Web-triggered outline import
## Context
Outline import currently only runs as a Claude Code subagent (`.claude/agents/outline-importer.md`) driven by a human typing `/import` in this CLI. James wants to kick off / resume the same import from the Novelly web app itself — a button on the Novel Overview tab and on the New Novel dialog, where the user pastes an absolute folder path (app is local-first; API and browser share a filesystem, confirmed with James). Parsing needs the same LLM judgment the subagent uses (title/blurb extraction, name-alias matching, arc-stage title synthesis, auto-creating characters mentioned in outlines without a dossier) — so this must run through the app's **embedded agent** (already wired to Anthropic via `ANTHROPIC_API_KEY`/`Agent:ApiKey`, same billing as existing agent features, no new credential).
Two capabilities don't exist yet and both are required:
1. **Filesystem access for the agent** — today's `NovelAgentToolset` is 100% DB-mediated; nothing reads files off disk.
2. **Background execution** — the existing agent endpoint (`POST /api/projects/{id}/agent/messages`) is fully synchronous and capped at 12 tool-call iterations per request. A 46-chapter import needs hundreds of tool calls over minutes — that has to run off the request thread, with a pollable status the UI can watch.
## Design
### New `Imports` feature folder (`src/Novelly.Api/Imports/`)
- **`ImportJob` entity** — `Id`, `SourceRoot`, `ProjectId?`, `Status` (`Pending|Running|Completed|Failed|Paused`), `StatusMessage`, `ChaptersCompleted`, `ChaptersTotal`, `CreatedAt`, `UpdatedAt`. New DbSet on `NovelDbContext`, new EF migration (`dotnet ef migrations add AddImportJobs`).
- **`ImportService`**:
- `InspectAsync(sourceRoot)` — read-only. Validates the path exists/is a directory, looks for `<sourceRoot>/.novelly-import.json`, reports one of: *no ledger* (fresh), *ledger incomplete* (resumable, with counts), *ledger complete* (all chapters + passes done). This backs the "check if there is work to do" requirement before anything starts.
- `StartOrResumeAsync(sourceRoot, forceRestart)` — fresh/resume: create-or-reuse an `ImportJob` row, enqueue its id, return the job. `forceRestart` (used for "complete → delete and reimport"): deletes the ledger file **and** the project the ledger points at (via existing `ProjectService.DeleteAsync` — cascades chapters/characters/beats like any other project delete), then starts clean. This is destructive — the UI must confirm before sending `forceRestart: true`.
- `GetStatusAsync(jobId)` — for polling.
- **`ImportEndpoints`**: `POST /api/imports/inspect`, `POST /api/imports` (`{ sourceRoot, forceRestart? }`), `GET /api/imports/{id}`. Same `RequestLoggingEndpointFilter`/`ValidationEndpointFilter` pattern as every other `MapGroup`.
- **`ImportJobRunner : BackgroundService`** — the only background-job infra in the app; there's currently none (no Hangfire/Quartz/hosted workers), so this is new. Backed by a singleton `Channel<Guid>`. `ImportService` writes job ids to the channel; the runner dequeues, opens a DI scope, resolves `ImportAgentService`, runs the import, updates the `ImportJob` row (progress + terminal status), catches exceptions into `Status = Failed`.
### New `ImportAgentToolset` + `ImportAgentService` (mirrors `NovelAgentService`/`NovelAgentToolset`, not extends)
Kept separate from the chat agent's toolset deliberately — filesystem access must never leak into normal chat conversations.
- **Filesystem tools, root-scoped to one validated `sourceRoot` per run** (constructed per job, not shared DI singleton):
- `list_source_files(pattern)` — enumerate under root only.
- `read_source_file(relativePath)` — read text; canonicalize and reject anything resolving outside root (path traversal guard); size cap.
- `read_ledger()` / `write_ledger(json)` — the **only** write capability given to this agent, scoped to exactly `.novelly-import.json`. This enforces "never write anything under the source root except the ledger" at the tool-permission layer instead of trusting prompt text alone (matches the "promote to a dedicated tool for a security boundary" principle).
- **Domain tools**: same application services `NovelAgentToolset` already wraps (`ProjectService`, `CharacterService`, `CharacterArcService`, `ChapterService`, `BeatService`) — reuse those services, add thin tool wrappers for `create_project`, `update_project_brief`, `create_character`, `update_character`, `add_arc_stage`, `create_chapter`, `update_chapter`, `create_beat`. Same architecture rule as everywhere else: new capability = same service method, just a different surface.
- **System prompt**: the outline-importer's passes (preflight → project → characters (with auto-create for undossiered names, per the just-shipped agent update) → chapters+beats → arcs), ported from `.claude/agents/outline-importer.md` into a template string, since there's no separate "subagent instructions" concept server-side — it's one system prompt.
- **Multi-turn driving loop** (this is what replaces the CLI's "batch 10 chapters, human re-invokes" pattern): each call to the model loop runs with a higher `MaxIterations` (e.g. 40) than the chat agent's 12. After the model stops calling tools, the **runner** — not the model — reads the ledger file directly and checks ground truth: are all chapters in `completedChapters`, are `completedPasses` complete? If not, send a synthetic "Continue the import from the ledger" user turn and loop again. Cap total driver-turns/tool-calls as a cost safety net; hitting the cap without completion sets `Status = Paused` (not `Failed`) with a message telling the user calling start again will resume — same ledger-driven resumability the CLI subagent already has.
### Web client (`src/Novelly.Web`)
- `api/types.ts` — add `ImportJob`, `ImportInspection`.
- `api/hooks.ts``useInspectImport()`, `useStartImport()`, `useImportJob(jobId)` (poll via `refetchInterval`, stop on terminal status — this is the first polling pattern in the app; no SSE/websocket infra exists to reuse), `useRestartImport()`.
- New `components/ImportDialog.tsx`: text input for the absolute path → "Check" calls inspect → renders one of: *Start import* (fresh) / *Resume import* (incomplete, shows counts) / *Import complete — Delete & reimport* (complete, behind a `confirm()`-style guard same as the existing project-delete Danger Zone button). Once a job is running, show progress (`Spinner` + chapters completed/total, reusing the existing progress-bar markup from `OverviewPage.tsx`'s Progress card) via the polling hook; on completion, invalidate project/chapter/character queries.
- Wire the dialog into two places:
- `OverviewPage.tsx` — new aside card ("Import outline") between the existing Progress and Danger zone cards, same `card p-5` shape.
- `ProjectsPage.tsx`'s `CreateProjectModal` area — a secondary "Import from outline" button next to Create, since import mints its own project and bypasses the manual create form.
## Verification
- `dotnet test` — add `ImportServiceTests` (inspect: no ledger / incomplete / complete; start enqueues a job; forceRestart deletes project + ledger) and `ImportAgentToolsetTests` (path-traversal rejection on `read_source_file`, ledger-only write scope) following the existing `ServiceTestFixture`/`ScriptedModelClient` patterns — model calls stay faked, never hit the real Anthropic API from tests.
- Run the app for real per CLAUDE.md ("build+tests passing ≠ working" — this touches an endpoint and a background loop): `dotnet run --project src/Novelly.AppHost`, use the new Overview card against `examples/blade-itself` (already has a partial ledger from the CLI run) to confirm resume works, then against a fresh folder to confirm a clean run, then re-run against the completed one to confirm the delete-and-reimport path.