novelly/docs/plans/api/outline_import_agent_plan.md

Web-triggered outline import

Context

Outline import currently only runs as a Claude Code subagent (.claude/agents/outline-importer.md) driven by a human typing /import in this CLI. James wants to kick off / resume the same import from the Novelly web app itself — a button on the Novel Overview tab and on the New Novel dialog, where the user pastes an absolute folder path (app is local-first; API and browser share a filesystem, confirmed with James). Parsing needs the same LLM judgment the subagent uses (title/blurb extraction, name-alias matching, arc-stage title synthesis, auto-creating characters mentioned in outlines without a dossier) — so this must run through the app's embedded agent (already wired to Anthropic via ANTHROPIC_API_KEY/Agent:ApiKey, same billing as existing agent features, no new credential).

Two capabilities don't exist yet and both are required:

1. Filesystem access for the agent — today's NovelAgentToolset is 100% DB-mediated; nothing reads files off disk.
2. Background execution — the existing agent endpoint (POST /api/projects/{id}/agent/messages) is fully synchronous and capped at 12 tool-call iterations per request. A 46-chapter import needs hundreds of tool calls over minutes — that has to run off the request thread, with a pollable status the UI can watch.

Design

New Imports feature folder (src/Novelly.Api/Imports/)

• ImportJob entity — Id, SourceRoot, ProjectId?, Status (Pending|Running|Completed|Failed|Paused), StatusMessage, ChaptersCompleted, ChaptersTotal, CreatedAt, UpdatedAt. New DbSet on NovelDbContext, new EF migration (dotnet ef migrations add AddImportJobs).
• ImportService:
  • InspectAsync(sourceRoot) — read-only. Validates the path exists/is a directory, looks for <sourceRoot>/.novelly-import.json, reports one of: no ledger (fresh), ledger incomplete (resumable, with counts), ledger complete (all chapters + passes done). This backs the "check if there is work to do" requirement before anything starts.
  • StartOrResumeAsync(sourceRoot, forceRestart) — fresh/resume: create-or-reuse an ImportJob row, enqueue its id, return the job. forceRestart (used for "complete → delete and reimport"): deletes the ledger file and the project the ledger points at (via existing ProjectService.DeleteAsync — cascades chapters/characters/beats like any other project delete), then starts clean. This is destructive — the UI must confirm before sending forceRestart: true.
  • GetStatusAsync(jobId) — for polling.
• ImportEndpoints: POST /api/imports/inspect, POST /api/imports ({ sourceRoot, forceRestart? }), GET /api/imports/{id}. Same RequestLoggingEndpointFilter/ValidationEndpointFilter pattern as every other MapGroup.
• ImportJobRunner : BackgroundService — the only background-job infra in the app; there's currently none (no Hangfire/Quartz/hosted workers), so this is new. Backed by a singleton Channel<Guid>. ImportService writes job ids to the channel; the runner dequeues, opens a DI scope, resolves ImportAgentService, runs the import, updates the ImportJob row (progress + terminal status), catches exceptions into Status = Failed.

New ImportAgentToolset + ImportAgentService (mirrors NovelAgentService/NovelAgentToolset, not extends)

Kept separate from the chat agent's toolset deliberately — filesystem access must never leak into normal chat conversations.

• Filesystem tools, root-scoped to one validated sourceRoot per run (constructed per job, not shared DI singleton):
  • list_source_files(pattern) — enumerate under root only.
  • read_source_file(relativePath) — read text; canonicalize and reject anything resolving outside root (path traversal guard); size cap.
  • read_ledger() / write_ledger(json) — the only write capability given to this agent, scoped to exactly .novelly-import.json. This enforces "never write anything under the source root except the ledger" at the tool-permission layer instead of trusting prompt text alone (matches the "promote to a dedicated tool for a security boundary" principle).
• Domain tools: same application services NovelAgentToolset already wraps (ProjectService, CharacterService, CharacterArcService, ChapterService, BeatService) — reuse those services, add thin tool wrappers for create_project, update_project_brief, create_character, update_character, add_arc_stage, create_chapter, update_chapter, create_beat. Same architecture rule as everywhere else: new capability = same service method, just a different surface.
• System prompt: the outline-importer's passes (preflight → project → characters (with auto-create for undossiered names, per the just-shipped agent update) → chapters+beats → arcs), ported from .claude/agents/outline-importer.md into a template string, since there's no separate "subagent instructions" concept server-side — it's one system prompt.
• Multi-turn driving loop (this is what replaces the CLI's "batch 10 chapters, human re-invokes" pattern): each call to the model loop runs with a higher MaxIterations (e.g. 40) than the chat agent's 12. After the model stops calling tools, the runner — not the model — reads the ledger file directly and checks ground truth: are all chapters in completedChapters, are completedPasses complete? If not, send a synthetic "Continue the import from the ledger" user turn and loop again. Cap total driver-turns/tool-calls as a cost safety net; hitting the cap without completion sets Status = Paused (not Failed) with a message telling the user calling start again will resume — same ledger-driven resumability the CLI subagent already has.

Web client (src/Novelly.Web)

• api/types.ts — add ImportJob, ImportInspection.
• api/hooks.ts — useInspectImport(), useStartImport(), useImportJob(jobId) (poll via refetchInterval, stop on terminal status — this is the first polling pattern in the app; no SSE/websocket infra exists to reuse), useRestartImport().
• New components/ImportDialog.tsx: text input for the absolute path → "Check" calls inspect → renders one of: Start import (fresh) / Resume import (incomplete, shows counts) / Import complete — Delete & reimport (complete, behind a confirm()-style guard same as the existing project-delete Danger Zone button). Once a job is running, show progress (Spinner + chapters completed/total, reusing the existing progress-bar markup from OverviewPage.tsx's Progress card) via the polling hook; on completion, invalidate project/chapter/character queries.
• Wire the dialog into two places:
  • OverviewPage.tsx — new aside card ("Import outline") between the existing Progress and Danger zone cards, same card p-5 shape.
  • ProjectsPage.tsx's CreateProjectModal area — a secondary "Import from outline" button next to Create, since import mints its own project and bypasses the manual create form.

Verification

• dotnet test — add ImportServiceTests (inspect: no ledger / incomplete / complete; start enqueues a job; forceRestart deletes project + ledger) and ImportAgentToolsetTests (path-traversal rejection on read_source_file, ledger-only write scope) following the existing ServiceTestFixture/ScriptedModelClient patterns — model calls stay faked, never hit the real Anthropic API from tests.
• Run the app for real per CLAUDE.md ("build+tests passing ≠ working" — this touches an endpoint and a background loop): dotnet run --project src/Novelly.AppHost, use the new Overview card against examples/blade-itself (already has a partial ledger from the CLI run) to confirm resume works, then against a fresh folder to confirm a clean run, then re-run against the completed one to confirm the delete-and-reimport path.