Documentation indexing

Core owns product documentation search in src/search/docsSearchService.ts. src/brain/tools/docsSearchTool.ts defines the built-in DocsSearch tool. The daemon injects its canonical bundled-first plugin scan roots, the shared SemanticVectorStore, the existing EmbeddingService and live embedding configuration; files in plugin documentation trees are read as data, never imported. The same service is registered through PluginRegistry.registerHostControl('docs') for the command palette and declared plugin consumers, and it is the docs corpus source the embed queue indexes.

resolveDocsRoot finds the numbered manual next to the package. readCorpus reads Markdown pages whose filenames start with at least two digits and a hyphen under docs/site (including three-digit article numbers and .cs.md / .sk.md variants), including nested topics, plus the curated docs/modules and docs/plugin-author trees. The package explicitly ships those two developer trees. Private plans, reviews, tools notes, audit files and top-level operator references are outside that allowlist. Missing curated trees are an installation error, not an empty corpus.

readPluginDocs walks each manifested plugin's docs/ directory recursively, preserves the full citation path and follows the first-root-wins identity rule. Installed plugins remain readable when disabled; an absent scan root or a plugin without documentation contributes nothing. Existing documentation with invalid or missing metadata is different: parsing rejects the read and emits search.docs_frontmatter_invalid, never silently omits the page or returns an empty successful catalog. Core and bundled plugin articles require the authored schema. Installed third-party sources instead receive one source-owned metadata projection: docs/ is plugin-reference / user and docs/developer/ is plugin-development / developer, independently of historical display-value group strings. Their authored title, slug and order remain required; section identities come from the same shared heading tree, including explicit mappings when supplied. This projection does not rewrite installed files or interpret old group labels as a second category format. Files remain data, never executable plugin imports.

The shared packages/plugin-shared/document.mjs parser owns YAML validation, Markdown tokens, headings, stable section IDs, links and image references. parseCorpus adds source ownership and category membership, requires authored article identity and, for core/bundled sources, a section mapping for every heading, groups .cs.md / .sk.md siblings with their English original and validates translation structure. Slug, audience, group, order and feature ownership remain canonical English metadata. User reads exclude developer sources (docs/modules, docs/plugin-author, plugin docs/developer); administrator guides remain readable to everyone with an audience label. Missing translations resolve to the complete English variant, reported as locale:'en', translated:false, never mixed paragraphs.

Derived section IDs are ${slug}:${locale}#${sectionId}:${part}. Stable authored anchors survive heading reorder and translation; split-part positions are internal, never public section links. The vector remains addressed only by the exact ${docTitle} › ${heading}\n${body} text hash and active vector space. All variants use the existing docs corpus in semantic_vectors; there is no Guides index, migration or query-time document embedding. The daemon queue fills missing hashes and prunes obsolete membership on its next drain. Restart discards only derived in-memory caches and reuses matching vector hashes; see the semantic index.

DocsLookupControl and the importless DTOs in src/shared/docs.ts expose search(query, limit, context?, execution?), catalog(locale), article(slug, locale) and asset(slug, assetId). Search defaults to English/all for DocsSearch; Guides and both Ask AI entry points explicitly request {locale, projection:'user', group?}. Passages add articleId, sectionId, actual locale and owner-resolved href to their citation path/title/heading/text. User links open /guides/<slug>#<sectionId>; developer citations open the public reference instead. Catalogs contain localized categories from the shared authored category module and ordered user/admin article metadata, not bodies. Articles add sanitized HTML, TOC, same-category previous/next and related user guides. A missing article or unreferenced asset returns null; source/read/validation failures reject.

The server-only DocsSearchExecution fourth argument carries {userId, signal?} independently of the retrieval DTO. Omission inherits the active tool contribution account; explicit null is instance work. HTTP supplies the authenticated account, never a body-supplied payer, so query embeddings use the existing spend admission and account usage sink. DocsSearch passes its contribution account and native tool cancellation signal. Both HTTP consumers combine caller cancellation with a five-second retrieval lifetime and abort outstanding provider work when that lifetime ends. Account refusals and cancellation propagate instead of becoming successful lexical results; HTTP preserves the shared spend-limit/accounting errors and caller-aborted 499 response. No turn/session state is fabricated.

Bodies are cached by source/citation path, inode/device, mtime, ctime and size. Every operation rescans membership and metadata, invalidating additions, edits, removals, renamed files and plugin uninstall. Parsed documents and immutable chunk arrays are reused until that state changes; warm BM25 calls segment only the query. Catalog revision hashes every source including metadata-only changes, shared categories and referenced image content hashes. Assets are separately revalidated so replacing an image also changes the revision and its URL without editing Markdown. Removed files/references retire their caches; no stale content-addressed asset URL is served.

Search uses the parser's heading sections and the existing 1,500-character paragraph-splitting policy. BM25 uses whole-word Intl.Segmenter, Unicode decomposition/diacritic normalization, tested en/cs/sk stop words and threefold heading weighting. Retrieval ranks the requested language with canonical English for product-name/cross-language matching, resolves equivalent sections to the display variant and deduplicates by article after either lexical ranking or hybrid RRF, before applying the limit. Hybrid ranks only vectors in the active space and reports coverage for that retrieval projection. Missing embeddings stay keyword-only; ordinary query-provider failure retains lexical results with its explicit reason, while account refusals and cancellation reject; vector-store failure rejects. Empty matched results mean no match, never a missing corpus. Scores are ranking values, not confidence.

Source reads reject symlinks and root escapes, check the opened regular-file descriptor and cap Markdown at 1 MiB and images at 10 MiB. Only parser-discovered local PNG/JPEG/GIF/WebP references with nonempty alt text enter the article asset map; extension and byte signature must agree. The asset ID is its SHA-256 plus extension, never an arbitrary filesystem path. Remote images and executable SVG are refused, not fetched. Shared renderDocument uses omitTitleHeading:true because the Guides header owns the single H1; the title's stable anchor survives as a non-heading span and nested headings/TOC remain unchanged. The Node sanitizeDocumentHtml boundary runs last. Link adapters resolve known canonical slugs, source-relative Markdown files and stable anchors; existing application /settings deep links pass through unchanged. An existing non-indexed regular file inside its owning plugin package (for example ../NOTICE) deliberately renders as its formatted label only, through resolveLink:null; link-only validation uses the plugin root and never reads or serves the file's bytes. This does not widen the image boundary beyond plugin docs/. Missing targets, directories, symlinks and package escapes still reject article rendering; unknown slugs never become public-URL fallbacks. Catalog/body/asset reads perform no inference. Each configured search embeds only its query; indexing cost belongs to the existing background queue.

DOCS_SEARCH_TOOL in src/shared/coreToolNames.ts is the account-grant name consumed by the tool factory. The standard session capability group, read-only agent catalogue (which plan mode also admits) and built-in catalogue expose it to account grants and deferral settings. Administrators hold the same finite tool grants as other accounts. The name is one of the core grantable names coreGrantableToolNames() returns, so fullAdministratorGrant (src/shared/toolGrants.ts) puts it in the grant an instance's first administrator receives; every other account reaches it only when its own elowen.users.allowed_tools lists it. That column defaults to empty, so a new account holds no tool until an administrator grants one, and a chatbot account is created with the chatbot plugin granted and no tools. No migration widens an existing account's grant; an administrator opens it from the Users panel or an account preset.

Core derives the corpus from the shipped documentation trees and stores only vectors: there is no documentation plugin and no plugins-data index behind the search.

Real consumers are Guides, the command palette's explicit Ask AI action and the DocsSearch tool. Regression coverage is in tests/brain/tools/docsSearchTool.test.ts, tests/brain/tools/docsSearchPluginCorpus.test.ts, tests/search/docsCorpusCache.test.ts, tests/search/docsLocalizedCorpus.test.ts, tests/search/docsCorpusContent.test.ts and tests/embeddings/semanticRank.test.ts. The store and its schema are covered by packages/db/tests/search/semanticVectorStore.test.ts and packages/db/tests/search/semanticVectorMigration.test.ts. The seam catalog remains authoritative.

The HTTP adapter in src/api/routes/docs.ts resolves the current docs control per request through the normal authenticated route group, with existing licence gates unchanged. Catalog and article reads require an explicit locale=cs|sk|en and return the shared DocsCatalog / DocsArticle DTOs without transport-specific copies. Audience is a reader label: ordinary authenticated members may read administrator-labelled guides. No control or failed source read answers 503 docs-unavailable; a valid empty catalog remains 200, while missing articles and assets answer 404 article-not-found / asset-not-found. Invalid identities or locales answer 400 before invoking the control. Catalog and article responses use no-store, leaving derived query caching to the browser's existing query owner.

Asset requests pass only a bounded canonical article slug and opaque image id to docs.asset. The corpus owns file containment, references and allowed image types; HTTP never accepts a filesystem path. It streams exact bytes with the corpus content type, nosniff and a quoted content-hash ETag. private, no-cache permits byte reuse only after revalidation: the control resolves current membership before returning 304, so a removed image/plugin does not retain an immutable browser grant. Source failures remain 503 rather than missing images.

The same-origin catch-all BFF streams these resources using its existing cookie-to-bearer authentication, path-segment validation and disconnect signal. Its shared header allowlist carries If-None-Match, and response ETags/cache headers survive unchanged. Documentation POST search joins the existing bounded-read deadline; it does not add another proxy. elowenClient.docsCatalog(locale, signal?), docsArticle(slug, locale, signal?), docsAssetUrl(slug, assetId) and docsSearch(query, locale, { limit?, group? }, signal?) are the browser seam. They load metadata or one article on demand, never preload a second corpus, and preserve typed failures instead of replacing them with empty arrays. The Guides module and palette are consumers; UI query lifecycle remains outside this transport owner. See API search semantics for bounds, budgets, Ask AI citation validation and error codes. These adapters add no persistent state or vectors and require a matched daemon/web release. Tests are tests/api/docsSearchExecution.test.ts (real route, docs service, embedding service, spend gate and usage sink with fake provider transport), tests/api/searchRoutes.test.ts, tests/search/siteSearchAsk.test.ts, web/tests/app/api/proxy.test.ts and web/tests/lib/elowenClient.test.ts.