Add outline import feature; drop Dto naming, map entities at the API boundary
Services now return entities; endpoints (and the agent toolsets) map to *Response records instead of services building wire DTOs themselves. Also brings in the outline-import agent, MCP tool, ledger and web dialog that were already in progress on disk.
This commit is contained in:
James Wampler
@@ -0,0 +1,25 @@
|
||||
# Web-triggered outline import — implementation summary
|
||||
|
||||
Implements the plan in `outline_import_agent_plan.md`.
|
||||
|
||||
## Backend (`src/Novelly.Api/Imports/`)
|
||||
|
||||
- `ImportJob` (+ EF migration `AddImportJobs`), `ImportPaths` (ledger read/write, root-containment path resolution, chapter-file counting — shared by the service and the toolset), `ImportDtos`, `ImportService`, `ImportEndpoints` (`POST /api/imports/inspect`, `POST /api/imports`, `GET /api/imports/{id}`).
|
||||
- `ImportAgentToolset` — root-scoped `list_source_files`/`read_source_file`/`read_ledger`/`write_ledger` (the only write capability, enforced at the tool layer, not just the prompt) plus `create_project`/`update_project_brief`/`create_character`/`update_character`/`create_chapter`/`update_chapter`/`create_beat`/`add_arc_stage` wrapping the same application services the REST API and chat agent use.
|
||||
- `ImportAgentService` — system prompt ported from `.claude/agents/outline-importer.md`'s passes; runs bounded turns (`AgentOptions.ImportMaxIterationsPerTurn` = 40 tool calls/turn, `ImportMaxTurns` = 8 turns), re-reading the ledger after each turn as ground truth for "done" rather than trusting the model.
|
||||
- `ImportJobRunner : BackgroundService` — the app's first background-job infra, a `Channel<Guid>`-backed queue drained in its own DI scope per job.
|
||||
|
||||
Wired into `NovellyServiceRegistration`/`Program.cs`. All 98 backend tests pass, including 14 new ones (`ImportServiceTests`, `ImportAgentToolsetTests`) covering inspect states, job dedup, force-restart's project+ledger deletion, path-traversal rejection, and the ledger-only write scope.
|
||||
|
||||
## Frontend (`src/Novelly.Web`)
|
||||
|
||||
- `api/types.ts` / `api/hooks.ts`: `ImportJob`/`ImportInspection` types, `useInspectImport`/`useStartImport`/`useImportJob` (polling, stops on terminal status).
|
||||
- `components/ImportDialog.tsx`: path input → Check → Start/Resume/Delete-and-reimport → progress polling → done. Wired into `OverviewPage.tsx` (new aside card) and `ProjectsPage.tsx` (button next to "New novel").
|
||||
|
||||
**Scope note:** import always creates its own project (it never populates the project you're already viewing) — the Overview card's copy says so explicitly and navigates to the new project on completion, since threading an "import into this existing project" mode through the toolset/ledger format wasn't part of the approved plan.
|
||||
|
||||
## Verified live (not just tests)
|
||||
|
||||
- Restarted the Aspire AppHost to pick up the migration + new code.
|
||||
- `POST /api/imports/inspect` against the real `examples/blade-itself` folder (partially imported by the CLI subagent earlier this session) correctly reported `Resumable`, 5/46 chapters, `["project","characters"]` passes — matches the CLI's own ledger exactly, confirming ledger-format compatibility between the two entry points.
|
||||
- `POST /api/imports` + polling against a synthetic one-chapter outline exercised the full endpoint → queue → background-job → status-transition path for real over HTTP; no `ANTHROPIC_API_KEY` is configured in this environment, so it terminated as `Failed` with the expected `AgentNotConfiguredException` message rather than a real import — this is the correct behavior for an unconfigured key, but it means the model-driven happy path (actually calling Claude and writing chapters/characters) has **not** been verified live. That needs a configured key and is worth a manual pass before considering this done-done.
|
||||
@@ -0,0 +1,47 @@
|
||||
# Web-triggered outline import
|
||||
|
||||
## Context
|
||||
|
||||
Outline import currently only runs as a Claude Code subagent (`.claude/agents/outline-importer.md`) driven by a human typing `/import` in this CLI. James wants to kick off / resume the same import from the Novelly web app itself — a button on the Novel Overview tab and on the New Novel dialog, where the user pastes an absolute folder path (app is local-first; API and browser share a filesystem, confirmed with James). Parsing needs the same LLM judgment the subagent uses (title/blurb extraction, name-alias matching, arc-stage title synthesis, auto-creating characters mentioned in outlines without a dossier) — so this must run through the app's **embedded agent** (already wired to Anthropic via `ANTHROPIC_API_KEY`/`Agent:ApiKey`, same billing as existing agent features, no new credential).
|
||||
|
||||
Two capabilities don't exist yet and both are required:
|
||||
1. **Filesystem access for the agent** — today's `NovelAgentToolset` is 100% DB-mediated; nothing reads files off disk.
|
||||
2. **Background execution** — the existing agent endpoint (`POST /api/projects/{id}/agent/messages`) is fully synchronous and capped at 12 tool-call iterations per request. A 46-chapter import needs hundreds of tool calls over minutes — that has to run off the request thread, with a pollable status the UI can watch.
|
||||
|
||||
## Design
|
||||
|
||||
### New `Imports` feature folder (`src/Novelly.Api/Imports/`)
|
||||
|
||||
- **`ImportJob` entity** — `Id`, `SourceRoot`, `ProjectId?`, `Status` (`Pending|Running|Completed|Failed|Paused`), `StatusMessage`, `ChaptersCompleted`, `ChaptersTotal`, `CreatedAt`, `UpdatedAt`. New DbSet on `NovelDbContext`, new EF migration (`dotnet ef migrations add AddImportJobs`).
|
||||
- **`ImportService`**:
|
||||
- `InspectAsync(sourceRoot)` — read-only. Validates the path exists/is a directory, looks for `<sourceRoot>/.novelly-import.json`, reports one of: *no ledger* (fresh), *ledger incomplete* (resumable, with counts), *ledger complete* (all chapters + passes done). This backs the "check if there is work to do" requirement before anything starts.
|
||||
- `StartOrResumeAsync(sourceRoot, forceRestart)` — fresh/resume: create-or-reuse an `ImportJob` row, enqueue its id, return the job. `forceRestart` (used for "complete → delete and reimport"): deletes the ledger file **and** the project the ledger points at (via existing `ProjectService.DeleteAsync` — cascades chapters/characters/beats like any other project delete), then starts clean. This is destructive — the UI must confirm before sending `forceRestart: true`.
|
||||
- `GetStatusAsync(jobId)` — for polling.
|
||||
- **`ImportEndpoints`**: `POST /api/imports/inspect`, `POST /api/imports` (`{ sourceRoot, forceRestart? }`), `GET /api/imports/{id}`. Same `RequestLoggingEndpointFilter`/`ValidationEndpointFilter` pattern as every other `MapGroup`.
|
||||
- **`ImportJobRunner : BackgroundService`** — the only background-job infra in the app; there's currently none (no Hangfire/Quartz/hosted workers), so this is new. Backed by a singleton `Channel<Guid>`. `ImportService` writes job ids to the channel; the runner dequeues, opens a DI scope, resolves `ImportAgentService`, runs the import, updates the `ImportJob` row (progress + terminal status), catches exceptions into `Status = Failed`.
|
||||
|
||||
### New `ImportAgentToolset` + `ImportAgentService` (mirrors `NovelAgentService`/`NovelAgentToolset`, not extends)
|
||||
|
||||
Kept separate from the chat agent's toolset deliberately — filesystem access must never leak into normal chat conversations.
|
||||
|
||||
- **Filesystem tools, root-scoped to one validated `sourceRoot` per run** (constructed per job, not shared DI singleton):
|
||||
- `list_source_files(pattern)` — enumerate under root only.
|
||||
- `read_source_file(relativePath)` — read text; canonicalize and reject anything resolving outside root (path traversal guard); size cap.
|
||||
- `read_ledger()` / `write_ledger(json)` — the **only** write capability given to this agent, scoped to exactly `.novelly-import.json`. This enforces "never write anything under the source root except the ledger" at the tool-permission layer instead of trusting prompt text alone (matches the "promote to a dedicated tool for a security boundary" principle).
|
||||
- **Domain tools**: same application services `NovelAgentToolset` already wraps (`ProjectService`, `CharacterService`, `CharacterArcService`, `ChapterService`, `BeatService`) — reuse those services, add thin tool wrappers for `create_project`, `update_project_brief`, `create_character`, `update_character`, `add_arc_stage`, `create_chapter`, `update_chapter`, `create_beat`. Same architecture rule as everywhere else: new capability = same service method, just a different surface.
|
||||
- **System prompt**: the outline-importer's passes (preflight → project → characters (with auto-create for undossiered names, per the just-shipped agent update) → chapters+beats → arcs), ported from `.claude/agents/outline-importer.md` into a template string, since there's no separate "subagent instructions" concept server-side — it's one system prompt.
|
||||
- **Multi-turn driving loop** (this is what replaces the CLI's "batch 10 chapters, human re-invokes" pattern): each call to the model loop runs with a higher `MaxIterations` (e.g. 40) than the chat agent's 12. After the model stops calling tools, the **runner** — not the model — reads the ledger file directly and checks ground truth: are all chapters in `completedChapters`, are `completedPasses` complete? If not, send a synthetic "Continue the import from the ledger" user turn and loop again. Cap total driver-turns/tool-calls as a cost safety net; hitting the cap without completion sets `Status = Paused` (not `Failed`) with a message telling the user calling start again will resume — same ledger-driven resumability the CLI subagent already has.
|
||||
|
||||
### Web client (`src/Novelly.Web`)
|
||||
|
||||
- `api/types.ts` — add `ImportJob`, `ImportInspection`.
|
||||
- `api/hooks.ts` — `useInspectImport()`, `useStartImport()`, `useImportJob(jobId)` (poll via `refetchInterval`, stop on terminal status — this is the first polling pattern in the app; no SSE/websocket infra exists to reuse), `useRestartImport()`.
|
||||
- New `components/ImportDialog.tsx`: text input for the absolute path → "Check" calls inspect → renders one of: *Start import* (fresh) / *Resume import* (incomplete, shows counts) / *Import complete — Delete & reimport* (complete, behind a `confirm()`-style guard same as the existing project-delete Danger Zone button). Once a job is running, show progress (`Spinner` + chapters completed/total, reusing the existing progress-bar markup from `OverviewPage.tsx`'s Progress card) via the polling hook; on completion, invalidate project/chapter/character queries.
|
||||
- Wire the dialog into two places:
|
||||
- `OverviewPage.tsx` — new aside card ("Import outline") between the existing Progress and Danger zone cards, same `card p-5` shape.
|
||||
- `ProjectsPage.tsx`'s `CreateProjectModal` area — a secondary "Import from outline" button next to Create, since import mints its own project and bypasses the manual create form.
|
||||
|
||||
## Verification
|
||||
|
||||
- `dotnet test` — add `ImportServiceTests` (inspect: no ledger / incomplete / complete; start enqueues a job; forceRestart deletes project + ledger) and `ImportAgentToolsetTests` (path-traversal rejection on `read_source_file`, ledger-only write scope) following the existing `ServiceTestFixture`/`ScriptedModelClient` patterns — model calls stay faked, never hit the real Anthropic API from tests.
|
||||
- Run the app for real per CLAUDE.md ("build+tests passing ≠ working" — this touches an endpoint and a background loop): `dotnet run --project src/Novelly.AppHost`, use the new Overview card against `examples/blade-itself` (already has a partial ledger from the CLI run) to confirm resume works, then against a fresh folder to confirm a clean run, then re-run against the completed one to confirm the delete-and-reimport path.
|
||||
Reference in New Issue
Block a user