Services and persistence

Services, persistence, and cleanup

Use ctx.dataDir() for plugin-owned writable files. The host chooses the persistent root and appends the plugin name. Do not infer another plugin's data directory or use it for Project files.

ctx.dataDir() creates the folder on the first call (src/plugins/pluginContext.ts, dataDir). A host that wires no data root makes the call throw rather than handing out a temporary path.

Use await ctx.db() only with reads: ["db"]. The host checks the grant and returns the plugin's PostgreSQL handle. Without that grant or a wired database, the call rejects. The host constructs this handle directly with createPluginDb from elowen-db; SQL, schema lifecycle and migration bookkeeping live in packages/db/src/pluginDb/index.ts.

const db = await ctx.db();
const schema = '"' + db.schema.replaceAll('"', '""') + '"';

await db.migrate([
  {
    version: 1,
    async up(tx) {
      await tx.run(
        `CREATE TABLE ${schema}.items ` +
        "(id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY, value text NOT NULL)"
      );
    }
  }
]);

The schema is plugin_ followed by the exact registered plugin name. Qualify every plugin object with the quoted db.schema; hyphenated names remain intact. A plugin name whose prefixed schema identifier would exceed 63 UTF-8 bytes is refused at handle creation, because PostgreSQL truncates longer identifiers. Values use PostgreSQL positional parameters. query returns all rows, one returns the first row or undefined, and run returns { rowCount }. Every operation is awaited. Do not add plugin-specific columns to core tables.

A database plugin also exports async function migrate(db) from its entry module. Put the existing db.migrate(steps) call there. The normal loader awaits that export before register; registration and runtime store initialization do not run migrations. A missing export or failed migration stops the plugin from loading. The export receives PluginDb<never> and cannot access host stores, runtime controls, services or configuration. Keep entry imports free of runtime effects and defer runtime-only imports until register. Bundled Sandbox and Changelog demonstrate this split.

Migrations run in version order only with daemon migration authority. Each awaited up(tx) and its ledger row commit together under the plugin's transaction advisory lock. Runner and worker migrate calls log a skip and resolve without applying steps. await db.appliedVersion() returns the highest committed version, or 0.

Use the transaction's supplied tx for every operation in an atomic unit:

await db.transaction(async tx => {
  await tx.run(`UPDATE ${schema}.items SET value = $1 WHERE id = $2`, [value, id]);
});

The promise resolves after commit; callback or commit failures reject and roll back. Handles and their methods expire when the callback finishes. A transaction can access the existing capability-filtered host repositories through tx.host; those operations share the same connection and commit. The root handle has no host accessor, and root SQL or host stores cannot be used inside the callback.

Before committing Project-owned state, call await tx.host.projects.requireActive(projectId) inside that same write transaction. It returns the active Project and holds its shared row lock until commit, conflicting with core deletion admission. A missing or deleting Project rejects; a root-store preflight alone cannot fence the final write. Sandbox uses this for execution leases and lifecycle admission; registry Sites uses it before committing publication state. Network work stays outside SQL. Cron and Sites may hold this lock across their existing atomic local file owner's final write only in a short, non-retrying READ COMMITTED transaction; filesystem failure propagates and no transaction retry repeats the effect.

For a coherent read-only snapshot, use the same transaction entry point with { isolation: 'repeatable read', readOnly: true }. Writes default to READ COMMITTED. A write that needs SERIALIZABLE passes { isolation: 'serializable', maxAttempts: 3 }. Eligible serialization failures and deadlocks retry the entire callback, so keep filesystem, network and runtime effects outside it.

Sandbox uses awaited transactions for lifecycle checkpoints, logs, execution leases and repository lease claim/repair/release. Stop timers and await pending heartbeats before final cleanup. Recheck ownership in the transaction and await the scoped DELETE. Pool connection acquisition is bounded to five seconds (connectionTimeoutMillis in packages/db/src/database.ts); slow SQL timing belongs to the process-owned PostgreSQL health diagnostics. Bound mutation batches remain short even when a higher-level operation spans multiple calls.

Conversation JSON state

Use StateStore from elowen-plugin-shared/stateStore for adapter conversation settings:

await state.patch(conversationId, { model: selection });
const settings = state.get(conversationId);
await state.pruneConversationKeys(['retiredOverride']);

Both write methods return promises. One store instance serializes writes to its destination and coalesces waiting patches in invocation order into one writeJsonAtomicAsync batch. Each patch synchronously replaces the cache, so reads expose the latest requested state and a synchronous get/compute/patch sequence sees preceding requests while disk writes are pending. Each batch snapshots that cache and resolves its promises after atomic rename. Successful writes leave any newer cached requests intact. A failed write logs and rejects its batch and every still-pending patch, clears the queue and invalidates the cache: those patches depended on discarded state. The next read reloads the committed file; later writes can then succeed. Use one instance per file and await startup migrations and final writes before shutdown. The shared runControlCommand('new', binding) awaits the generation write before confirming the new conversation. Await the command; a failed write rejects it without sending a success reply. No helper work runs without a caller. Teams is a real consumer; JSON serialization remains linear in the whole state size per batch, but disk waits yield and bursts avoid repeated serialization. See the shared helper contract. The contract version is exported as PLUGIN_SHARED_API_VERSION; adapters must migrate every patch call and every pruneConversationKeys call to await and declare that exact version.

Shared bounded deletes

Import drainDbRows and yieldDbBatch from elowen-plugin-shared/dbBatches. drainDbRows(db, { schema, table, identity, predicate, args?, bytes, beforeDelete? }) returns Promise<number>. Supply raw schema/table/identity names separately; the helper quotes each identifier. The integer identity is explicit, for example row_id or id. Use trusted PostgreSQL SQL fragments and bind values through args with $1, $2.

Each db.transaction(async tx => ...) locks matching rows in identity order using FOR UPDATE, selects at most 64 rows and 256 KiB of estimated bytes, awaits beforeDelete(tx, ids) when supplied and deletes those identities through the same handle. One oversized row progresses alone. bytes includes fixed overhead and octet_length of nullable payloads with COALESCE; it must return a non-negative safe integer. Bodies are not loaded to budget a transaction.

Call outside any transaction. No callback runs for an empty batch. A callback or DELETE failure rolls back that batch; earlier commits remain. A real setImmediate follows each nonempty commit after releasing row locks. yieldDbBatch(): Promise<void> provides the same yield for other bounded loops. Cronjob uses transaction-local daily rollups; GitHub, OneDrive, Chatbot, Browser and Todo use the deletion loop. See the shared helper example.

Register registerBootReconcile(fn) for idempotent repair on every daemon start. Register registerService({ name, start, stop }) for host-managed background work. Services start after boot reconciliation and stop when the daemon shuts down. registerInterval(name, fn, ms) uses an unref'd timer and is cleared on stop. Return asynchronous tick work: pending ticks are skipped, and stop drains the current tick. One or two failed ticks in a row stay in the local log. A series is reported exactly once on its third consecutive failure. Further failures remain local; a successful tick logs recovery and resets the series. A delegated runner loads plugin tools but does not start plugin services.

Boot reconciles run sequentially in registration order, each with a 20 second budget. A throw or timeout is logged and raises an admin alert, but it does not stop the boot or the reconciles after it (src/plugins/serviceRunner.ts, runBootReconciles). The budget stops waiting without cancelling work. Return and await cleanup promises; retire scheduler inputs before the first await and keep later batches idempotent, as Cronjob does for missing accounts, because platforms may start while timed-out cleanup is still running. Without a registration, nothing runs at boot for that plugin. The sub-agent runner never runs reconciles. The sandbox plugin registers one as a real example (plugins/sandbox/index.mjs).

Services start in registration order after the reconciles. A start failure is logged and raises an alert; the remaining services still start. On shutdown, interval timers stop first, then services stop newest first, each with a 30 second grace period (src/plugins/serviceRunner.ts, startAll and shutdownAll). registerService refuses a registration with no name or with a start or stop that is not a function (src/plugins/pluginContext.ts, registerService).

A throwing or rejected interval tick is logged, and the interval keeps running. Shutdown waits for ticks already in flight (src/plugins/pluginContext.ts, registerInterval). A tick costs host time every ms milliseconds, not model calls, unless the tick starts a turn. The sandbox plugin registers a 10 second lease-reconcile interval as a real example (plugins/sandbox/index.mjs). A tick that usually finds nothing to do should establish that with a read before opening a db.transaction. Await the entire tick, including pruning, so overlap prevention and shutdown draining cover pending database operations. Sandbox uses transaction-local row locks for operation and resource ownership; read-only multi-query admission uses a REPEATABLE READ snapshot. Root filesystem downloads await their progress callback before continuing the stream, so durable operation progress finishes or rejects before artifact publication and cannot outlive its operation.

Register registerUserRemoved(fn) for every account-owned row, file, secret, or schedule. The account row still exists while the callback runs. Disabled plugins cannot receive the callback, so boot reconciliation must also make cleanup safe after re-enablement. Core verifies account process teardown and removes managed sessions before these callbacks run; a failing callback refuses account deletion so it can be retried. Terminal removes the account's remembered working directories and in-flight bookkeeping. Subagent removes retained jobs, workflow state and interrupted recovery journals by their exact account principal; late child events cannot recreate removed journals or publish completions. Other accounts remain untouched. No extra cleanup service or per-account storage copy is involved.

Registering this handler is mandatory for any plugin that stores per-user state. Nothing else reaps that state: the account id is never reused, so leftover files or rows stay on the instance forever (src/plugins/api.ts, registerUserRemoved). Core runs the handlers in registration order. A handler that throws is logged, the rest still run, and the deletion request returns an error while the account stays (src/api/routes/users.ts, account delete handler). The admin can repeat the request. The bundled changelog, MCP, sandbox, subagent and terminal plugins each register a handler as real examples (plugins/*/index.mjs).

Register registerProjectRemoved(fn) for Project-owned state. Make cleanup idempotent. Core runs the handler while the Project row still exists, before it removes that row itself, so the id is still known. A managed Project's deletion runs inside the environment provider instead, which removes the row through the same store seam without calling these handlers, so boot reconciliation must reap that state too. A throwing handler is logged and Project deletion continues (src/projects/projectService.ts, Project delete path).

Atomic scoped secret creation

await bag.getOrCreate(key, create) returns the encrypted value and CAS version, or creates it atomically in PostgreSQL. Minimal use:

const token = (await ctx.instanceSecrets().getOrCreate('workerControlToken', () =>
  randomBytes(32).toString('base64url'))).value;

The bag already fixes plugin and owner scope. The factory runs only when the key is absent and must synchronously return a string. An unused bag performs no write; unavailable storage, corruption, invalid values, factory failure and database errors propagate to the caller. Existing key/value validation and the 1 MiB value limit apply. Database-side uniqueness and a transaction protect creation; it adds no second secret authority or network work. Await the bag's get/has/set/delete operations too. Use asynchronous withLock for credential refresh, not this factory. Sites caches its workerControlToken with get and set in the registry; getOrCreate performs that read and create as one step.

Binary atomic writes

elowen-plugin-shared/atomicJson declares writeTextAtomic(file: string, bytes: string | Uint8Array, options?): void. Minimal use: mkdirSync(dirname(file), { recursive: true }); writeTextAtomic(file, imageBytes);

Strings and byte arrays use the same existing sibling staging file and rename algorithm. The helper creates no parent directory and performs no work when not called. Existing durable, mode, stagingMode, forceMode and maxBytes options retain their semantics. A failed write or rename throws and removes only staging created by that attempt; there is no success-shaped fallback.