Navigácia
ELOWEN · Dokumentácia

Posledná aktualizácia: 20. októbra 2026

Prechádzať dokumentáciu · Shared semantic index

Tento článok zatiaľ nie je preložený. Je dostupný v angličtine.

Developer reference

Shared semantic index

One durable store for every DERIVED embedding vector the daemon keeps, one identity for the vector space those vectors live in, one queue phase that fills them and one ranking API that reads them. The documentation index, the route catalog and the command palette's candidate cache share this seam instead of one table, one cache and one rebuild per consumer.

Boundary

The corpora here are DERIVED: a document is read back from its source (documentation files, the live route list, the palette candidates in the request) on every pass, only the vector is stored, and the vector is addressed by the hash of the text it was computed from. Nothing in this seam relates a vector to an account, mutates a stored document or carries plugin state.

Memory is deliberately NOT a corpus here. A memory's vector hangs on a mutable, account-owned row, its write is a CAS over ownership, status, body hash and space in one transaction, its reads are per user with shared pools and its deletion cascades through a foreign key; a content-addressed table would lose that binding or duplicate it. Memory keeps its own store and shares everything else: the queue (where it has precedence), the pipeline, the space identity, cosine and the relevance floor. Its ranking with importance weights and paraphrase dedup stays its own. See Memory lifecycle and delivery.

The document model

SemanticDocument, CorpusSource and SEMANTIC_RELATED_FLOOR live in src/embeddings/semanticTypes.ts. SemanticCorpus is the store's own union and is declared with the vector repository in packages/db/src/repositories/search.ts, exported by elowen-db.

  • SemanticDocument is { id, text }: a stable caller-owned id and the exact text the vector is computed from. text is the embeddable string, never a stored copy of it.
  • SemanticCorpus is 'docs' | 'api' | 'palette': the three corpora that share the store.
  • CorpusSource is { corpus: 'docs' | 'api'; documents(): Promise<readonly SemanticDocument[]> }: one corpus the embed queue can index on its own by reading its documents from their source. The palette is not a source, because its texts are known only to the browser and arrive with the request.

Each corpus defines its own ids and texts, and the text is what makes the vector:

  • docs: one document per bounded shared-parser section part and locale, id = ${slug}:${locale}#${sectionId}:${part} and text = ${docTitle} › ${heading}\n${body} (src/search/docsSearchCorpus.ts parseCorpus, src/search/docsSearchService.ts sectionText). The docs source is DocsSearchService itself.
  • api: one document per route the daemon serves, id = ${method} ${path} and text the route's own words joined in a fixed order: method, path, its declared summary, the owner, a plugin's manifest description and the field names of its declared body schema, cut to 1,000 characters (src/api/routeCatalog.ts routeDocument, collect, catalogDocuments). The owner is core or the plugin name. The api source is built in src/daemon/bootstrap.ts from app.routes and the live plugin registry, read lazily at each drain.
  • palette: one document per candidate the browser sends ({ id, text }), ranked and stored by src/search/siteSearchRank.ts.

The HASH RULE: a vector is addressed by hashText(document.text) (elowen-plugin-shared/contentHash, SHA-256 of the exact UTF-8 text) plus its vector space. A changed document is therefore a hash that is MISSING, never a row that has to be overwritten, and the caller computes the hash so the queue and a query derive byte-identical text for the same document. Verifying that the two sides agree is what the per-corpus id/text definitions above guarantee.

Storage

packages/db/src/search/vectors.ts owns the awaited repositories.search.vectors methods. The PostgreSQL table is defined in packages/db/migrations/0001_baseline.sql:

CREATE TABLE elowen.semantic_vectors (
  row_id bigint GENERATED ALWAYS AS IDENTITY UNIQUE,
  corpus text NOT NULL,            -- 'docs' | 'api' | 'palette'
  space text NOT NULL,             -- EmbeddingService.spaceId(cfg)
  text_hash text NOT NULL,         -- hashText(embedded text)
  vector bytea NOT NULL,           -- packed Float32
  created_at timestamptz NOT NULL DEFAULT CURRENT_TIMESTAMP,
  PRIMARY KEY (corpus, space, text_hash)
);

The table holds no text: the source stays the authority for the document and this table holds only the vector of the text text_hash names. space is the vector space identity (EmbeddingService.spaceId(cfg)), so switching the model, its endpoint or its requested width MISSES every row instead of ranking across incompatible spaces.

semantic_vector_state (corpus PRIMARY KEY, revision) carries one monotonic revision per corpus, advanced by an INSERT and a DELETE trigger on semantic_vectors. The three rows (docs, api, palette) are seeded by the baseline migration itself, so a projection read before the first index is never pinned at revision 0 forever. There is no UPDATE trigger on purpose: put inserts with ON CONFLICT DO NOTHING and a vector is never overwritten, so only an insert or a delete changes what a corpus holds. The per-corpus revision is what lets the committed projection of one corpus be cached across the writes of another: a palette write never invalidates the documentation projection.

All seven repositories.search.vectors methods return promises and are awaited by their real callers:

  • missing(corpus, space, hashes): keys with no vector, for the queue. Keys only, no BLOB is read.
  • get(corpus, space, hashes): stored vectors for a hash list, for the palette's get/put path.
  • vectors(corpus, space): the corpus's whole committed projection with unpacked vectors, cached per corpus and invalidated by that corpus's revision. Revision and rows are read in one transaction, so a commit from another process cannot split the two.
  • put(corpus, space, entries): bounded batches on the repository's PostgreSQL transaction.
  • prune(corpus, space, live): after docs or api completes, deletes the rows whose hash the finished corpus did not report as live. One pass commits only what fits a single payload budget and returns how many rows it removed, so its one caller repeats it until it returns 0.
  • pruneSpace(activeSpace): deletes every row of EVERY corpus whose space is not the active one. This is the only path that retires a dead vector space, because the palette's texts live in the browser and nothing can report them stale.
  • evictOldest('palette', max): FIFO eviction of the palette cache back to max, oldest first with row_id breaking equal created_at timestamps.

The vector space

EmbeddingService.spaceId(cfg) (src/embeddings/embeddingService.ts) is JSON.stringify(this.memorySpace(cfg)), the serialized { provider, model, dimensions } where provider is a SHA-256 of [providerId, endpoint, dimensions ?? null]. It is THE vector-space identity and the only one: a vector made under one provider, endpoint, model or requested width is a different space from a vector made under another and is re-embedded instead of being compared with it. Every caller that has to compare two spaces compares this string, never its own serialization of the parts. Memory retrieval shares the same identity through MemoryService.assertCurrentSpace.

embeddingSpace(config, endpoint) is the shared identity calculation for the configured space and an actual authorized batch endpoint. EmbeddingBatch(config, texts, userId, pinnedSpace) requires the run's captured space. Its production implementation, embedBatchThroughWorker, captures the space from the exact authorizedEndpoint handed to the child. After the worker's paid usage commits, it calls assertEmbeddingSpace(actualSpace, pinnedSpace) before returning any vectors. Comparing the actual dispatch preserves correctness even when configuration moves A-B-A while the batch is in flight; no per-batch live endpoint reread is needed.

The queue's memory and corpus phases and MemoryMaintenanceService all pass their pinned space to that batch boundary. A mismatch throws EmbeddingSpaceChangedError, ends the run through its existing failure reporting and leaves the affected documents pending, while retaining earlier valid batches. The check hashes bounded metadata once per batch and adds no database or provider request. The corpus phase separately checks the live space before retiring other spaces. There is no optional bypass.

The VALIDITY RULE follows from the address: a vector is valid exactly while its text hash AND its vector space both still match. No version, mtime or boot re-index enters it. A restart therefore costs one key lookup per corpus and no embedding call.

The queue's corpus phase

EmbeddingQueue (src/embeddings/embedQueue.ts) is the only indexer of this store. Its memory phase is unchanged; a second phase follows it in the same tick and indexes the corpora the daemon hands the drain.

  • Wiring is DAEMON-ONLY: src/daemon/bootstrap.ts builds the docs source (docsSearch) and the api source (catalogDocuments(app.routes, await pluginProvider.get())) after the server exists and passes them as semanticCorpora to createMaintenanceLoops, whose 30-second tick calls embedQueue.drain(corpora) (src/daemon/maintenance.ts). The sub-agent runner builds the same EmbeddingQueue but has no maintenance loop, so it never drains and only READS the store.
  • Deduplication by text: a corpus is reduced to one entry per hashText(document.text) before it is looked up, so two sections or two routes with identical text share one vector and one request slot.
  • Request caps: at most 50 documents and 60,000 characters of their text per request (CORPUS_DOCS_PER_REQUEST, CORPUS_CHARS_PER_REQUEST). These bound ONE request, because their reason is what an embeddings endpoint accepts, so corpus requests run back to back inside one drain; there is no per-drain batch cap.
  • MEMORY PRECEDENCE: before each request after the first, the phase re-checks whether an account has memory work waiting and ends the tick if so. A corpus therefore cannot push the next memory tick out by more than one request; the first request of a tick is deliberately not gated, or a refused body would starve the corpora forever. An account the spend gate refuses is likewise no reason to end a tick: its own bodies stay pending for its next turn while the corpora, which are the INSTANCE's work and ask no account's gate, still index.
  • Per-process digest: corpusDigest(sortedHashes, spaceId) is remembered once a process saw a corpus complete in that very space. It is a pure optimisation over missing() for the restart case, where every document is already indexed and the tick spends one key lookup per corpus and no request.
  • After a corpus ends the tick complete, prune(corpus, spaceId, liveHashes) is repeated until it reports nothing left, and only then is the digest remembered: one pass leaves the rows that did not fit its commit behind, and a remembered digest would stop every later pass for that corpus.
  • The drain always starts with pruneSpace(spaceId), which retires a space nothing mentions any more. A space that differs from the one the digests were recorded in forgets all of them first, because that same move just deleted the rows they describe: returning to an earlier space re-embeds the corpus instead of trusting a digest about rows that are gone.

Errors and per-document refusals are reported through ConsecutiveFailures under the code search.index_embed_failed (see Problem producer inventory).

Ranking

src/embeddings/semanticRank.ts is the ONE vector ranker of this surface:

  • rankByVector(query, docs, vectors): cosine (elowen-plugin-shared/vectors) per document that has a vector. A document without a vector is SKIPPED rather than scored 0, so a partially indexed corpus ranks what it has instead of padding the tail with misses. Sorted by score descending, ties keeping the caller's document order.
  • fuseRanks(lists): Reciprocal Rank Fusion with RRF_K = 60; an id's score is the sum over the lists that carry it of 1 / (RRF_K + rank + 1). This is how a corpus's existing lexical ranking is combined with its semantic one: an id near the top of either list is promoted, and an id both lists agree on outranks one only a single list placed first, with neither ranker's scores having to be commensurable.

SEMANTIC_RELATED_FLOOR = 0.3 (src/embeddings/semanticTypes.ts) is the single relevance threshold for treating two texts as related. rankByVector does not apply it; each consumer decides whether it wants it (the palette and the catalog do, documentation search returns its top k as ranked). Memory recall may override the floor from a Setting; search always uses the default.

Behaviour with no model, on a failure and on a partial index

  • No model configured: every consumer is explicitly keyword/lexical (mode: 'keyword' for DocsSearch, mode: 'lexical' for the catalog), and the queue no-ops, leaving the corpora unindexed.
  • Query embedding fails: the answer is still the marked lexical one, with the reason in details and a search.query_embed_failed warning through ConsecutiveFailures (one line per outage). Losing the provider must not lose the search, because BM25 over the same corpus still exists.
  • Partial index (first boot after an upgrade, a changed model): the answer is HYBRID over what has a vector plus lexical over everything, and it carries coverage: { embedded, total }. A query NEVER waits for a rebuild.

Instance accounting

Indexing a corpus is the instance's own work, not an account's. It is admitted and authorized with a null payer (validateWorkAccount(null) returns null, admit admits instance work and the endpoint is authorized as { kind: 'instance' }), and the usage row it leaves carries INSTANCE_USAGE_USER_ID = 0 (src/embeddings/embeddingUsage.ts), the instance owner's row. embedBatchThroughWorker maps null to that constant for the usage report (src/brain/embedBatchWorker.ts); admission and authorization never see 0, because validateWorkAccount(0) reads 0 as "not a paying account" and throws. The instance scope of the usage report counts the row and no account's limit does, because a personal read filters on user_id = ?. Plugin embeddings with no account (undefined) stay uncounted, because nobody said the instance pays for them.

Limits and cost

Numbers are read from the owners named above.

  • Write batching: at most 64 rows and 256 KiB per transaction (createVectorBinder.put and BATCH_BYTES in packages/db/src/search/vectors.ts); a single oversized row commits alone. prune, pruneSpace and evictOldest each commit one budgeted window and yield between windows.
  • Request caps: 50 documents and 60,000 characters per corpus request; the memory phase keeps its own per-drain caps of 50 bodies and 60,000 characters (DEFAULT_EMBEDS_PER_DRAIN, DEFAULT_CHARS_PER_DRAIN).
  • Palette cache: at most 5,000 rows (PALETTE_VECTOR_CACHE_MAX in src/search/siteSearchRank.ts); past it the oldest rows are evicted. Past that cap an eviction costs one re-embed and nothing else.
  • Vector size: a Float32 vector is dimensions × 4 bytes, so a 4,096-wide vector is 16 KiB and a 1,536-wide one is 6 KiB. The documentation index is a few thousand sections (about 2,900 in the live instance), so its rows are tens of megabytes at 4,096 dimensions and a few hundred palette rows are a rounding error beside them.
  • First full index: one request per 50 documents / 60,000 characters. The shipped documentation is about 3.3 M characters of section text, so the first pass is roughly 55 requests, each a background worker fork plus one provider call; the wall-clock cost is that many request latencies, and a memory-waiting instance paces it at one request per 30-second tick. Afterwards only changed sections and changed routes cost anything.

DocsSearch, the worked example

DocsSearchService (src/search/docsSearchService.ts) is the real consumer that shows the whole seam: it owns the docs corpus, implements CorpusSource.documents(), caches the corpus by mtime and, on a query, reads semantic_vectors through vectors('docs', spaceId(cfg)), ranks the sections with keywordRank and rankByVector and fuses them with fuseRanks. It never embeds the corpus and never waits for it: a query embeds itself once and ranks over whatever vectors the store holds, reporting mode ('keyword' | 'hybrid') and projection-specific coverage. The DocsSearch tool, Guides and the docs host control read the same service. Every locale remains in corpus docs; user/all/group filters are derived retrieval views, not additional indexes. Authored article/section identity replaces numeric section positions; existing matching text hashes remain reusable, and the queue reconciles changed membership without a database migration or deleting shared vector state. Metadata-only source edits change the documentation catalog revision without forcing an unchanged section vector to be re-embedded. See Documentation answering.