Problem reporting boundary

The complete producer inventory classifies every declared code and lists its reachable core and registry producers, including mixed retry/terminal rules and declarations without a current caller.

Audience before retry

Only defects in Elowen or publisher plugins belong on the licence server. User or instance conditions remain local, including unavailable models/accounts, spending limits, offline paired computers and incompatible customer plugins. Local logs and owner alerts remain available.

ProblemService.project is the single collection authority. It combines the exhaustive PROBLEM_AUDIENCE code catalog with proven publisher ownership and the native InstanceConditionError cause chain. The logger preserves the original Error as ephemeral local LogEvent.failure; reports project only redacted text and never serialize that Error. Producers must preserve typed causes through generic failure wrappers. Error messages and a bare Error type are not evidence for audience.

The complete audience table and outstanding producer splits documents every declared code. The intended publisher-only rule remains incomplete at mixed producers that flatten structured failures to text; their existing reports are preserved instead of silently suppressing software defects. That limitation is explicit in the table.

Delegated boot recovery and result delivery retain the original failure in LogEvent.failure, including recovery/progress aggregates. Settlement preserves causes through job reminders (including jobs lost to restart), shared drain outcomes and concurrent result turns: contextual Errors use native cause, and both drain causes and pending settlement failures stay in memory only. A typed quota refusal stays local; a programming or persistence error still reports, even when its text resembles a provider refusal. Boot reconciliation of API-token runs logs api.token_run_interrupted as local-only: interruption by restart is an instance lifecycle condition, while the store still records an error and unknown cost. This does not suppress the separate producer of a failed usage settlement.

Host conditions

HostConditionError from elowen-plugin-shared/errors marks a condition of the host itself rather than of Elowen or the customer's configuration: pool acquisition failure (DatabaseConnectionError, also behind the protected configuration wrapper) and event-loop severe stalls explained by measured scheduling gaps. The operator wants to know when the server is struggling, but not about a single blip. ProblemService.observe is the only place that decides: per problem identity (code, level, plugin, plugin version) it keeps at most the last three occurrence times; the third occurrence within five minutes adds a count of three with that event's sample, and each further occurrence while the condition keeps recurring inside the window adds one. After five quiet minutes the next occurrence starts over. Every occurrence is still logged locally.

The rule acts only on the typed cause in LogEvent.failure. Explicit local-only metadata still wins, which keeps the licence-report route from reporting its own outage. Emitters therefore must not derive local-only from a host condition: the shared turn runner marks only non-host instance causes local. A host condition under an instance-only code stays local. There is no timer or storage; a restart begins with empty windows. Consumer: api.database_unavailable in src/api/server.ts.

Attempt-based reporting

The retry owner decides persistence before sending metadata to the logger. The shared elowen-plugin-shared/consecutiveFailures helper caps a count per operation/resource key: fail(key) is true only on failure three; reset(key) follows successful completion. Every failure is logged, with local-only on all other attempts. The gate decides when a failure reaches the licence server, never what a report contains: the reported attempt carries the same redacted message sample as an immediate one (reporting: gate ? undefined : 'local-only'). Error text is never a key. Cancellation and skipped work neither count nor reset. The helper adds no timers or storage; durable recovery and delivery owners use their existing database attempt counters instead.

Periodic daemon maintenance uses one key per operation, with separate keys for account runtime preparation and personal catalog refresh. Session deletion drain and optional session retention share a code but have independent counters. Hourly sweeps therefore report on the third failed scheduled attempt, normally two hours after the first failure; daily memory retention and attachment sweeps normally report two days after the first. The activity snapshot cache counts actual refresh attempts; the checkpoint service counts database failures separately from worker startup/crash attempts. Local logs remain immediate throughout. A malformed attachment-worker reply reports immediately rather than waiting for later daily attempts.

The rule applies to retryable failures, not definite invalid state, one-shot requests, boot/update failures or exhausted recovery. runtime.reporting_invalid and plugin.reporting_invalid remain immediate. When measured profiler scheduling gaps explain the severe-stall threshold of runtime.event_loop_lag, the warning carries a host condition (below). The profiler intersects each sampled and unscheduled interval with the requested stall; the watchdog subtracts only that attributed unscheduled time, capped to each attributed stall, before applying the existing two-second reporting threshold; unprofiled time, sampled application work and sustained p99 saturation remain reportable. This uses numeric profiler evidence, never frame labels or message text. runtime.slow_step remains local, and a pending non-provider step reports runtime.stuck_step once on its third thirty-second observation. Provider request failures, retries and exhausted upstream refusals remain instance-local. Exhausting a retry budget does not turn an account condition into an Elowen defect.

Owning contracts and regression suites: provider maintenance, catalog attempts, memory embeddings, durable recovery, Sandbox sampling and alerts, and plugin interval ownership.

Daemon problem reporting boundary

Problem reporting is a core-owned licensing transport because customers do not install the operator's licensing plugin. Its contract is src/licence/contract.ts: codes follow the open subsystem.event grammar (PROBLEM_CODE_RE) rather than a closed enum, so a newer core with new codes never has its batches rejected by an older licensing server; the server describes the codes it knows and shows any other code verbatim. Core still emits only code-defined identifiers. Core-side emitters are checked against the closed PROBLEM_CODES catalog (src/licence/contract.ts, liveProblemMetadataSchema in src/problems/service.ts), and the wire grammar (PROBLEM_CODE_RE in src/licence/contract.ts) also admits unclassified for batches from older cores. The receiving side runs on the operator's licensing server, outside this repository, and owns storage and the 14-day retention. Core declares that window as PROBLEM_LIMITS.retentionMs in src/licence/contract.ts, but no core code reads it. Core sends the active licence token as a Bearer credential. The credential callback of the ProblemService construction in buildApp (src/daemon/bootstrap.ts) reads the token at send time and returns nothing unless the live licence state is active, so no report leaves an inactive licence. Each batch is one request with a 15-second deadline and no redirects (sendProblemBatch in src/licence/reports.ts). Core verifies the licence offline with its embedded public key (licencePublicKey in src/licence/publicKey.ts); checking the report credential is the receiver's job. Only redacted daemon warning/error samples enter this path. Only the daemon installs the log sink (setLogSink in buildApp, src/daemon/bootstrap.ts), so no agent, runner or web log text does. Each aggregate carries a code, a level, the plugin and its version, a count and at most three redacted single-line samples of up to 500 characters. reporting has a single value, local-only, which keeps a line on the instance; the central audience policy also excludes typed instance conditions, instance-only codes and customer-plugin events. Eligible reports carry their redacted message sample and any host resource snapshot the emitter attaches. Only an untyped producer's metadata fault (runtime.reporting_invalid, plugin.reporting_invalid) is counted without text, because that prose is not classified by any declared code. A batch holds at most 50 entries and 16 KiB. Batches are persisted in problem_outbox and leave it when the receiver accepts the batch or refuses it permanently (400, 413, 415 or 422). Other failures are retried, and a 401 or 403 stops sending with that token. The receiver must echo the batch id in its acknowledgement, and Retry-After is honoured up to 30 minutes. Disabling reporting deletes the outbox. If that purge fails, it stays mandatory and runs again before any later enable.

For core runtime.event_loop_lag entries only (plugin: 'none'), optional hostResources is validated by problemHostResourcesSchema in the same contract. It carries numeric sampledAt in epoch milliseconds, cpuPercent, ram and disk pairs of usedBytes/totalBytes, io.readMBps/writeMBps in decimal MB/s, and network.rxMbps/txMbps in decimal Mb/s. Unreadable resource groups are omitted; zero is a real reading. Strict nested objects refuse text and unknown keys, CPU is bounded to 0–100, quantities to Number.MAX_SAFE_INTEGER, capacities require a positive total and used no larger than total, and the timestamp must be a nonnegative integer within the JavaScript Date range. The existing entry and 16 KiB batch limits still apply. The browser mirror in web/lib/problemReporting.ts carries the identical validation block for Settings' last-sent inspection. The opt-in notice names the host sample in cs/sk/en.

The minute sampler in src/hostMetrics/sampler.ts returns this projection through the existing checkpoint worker maintenance reply, after storing and pruning its sample. buildApp caches it; watchLoopLag reads that cache once when emitting a warning. No host reads, SQL, extra worker or timer run on the stalled warning path. Before the first successful sample, the optional field is absent. The sample timestamp remains explicit because the last completed minute sample can predate the warning, especially while maintenance is delayed. A minute aggregate keeps its latest attached snapshot, while the existing bounded text samples retain stall duration and blocked by: attribution without a second attribution field.

The licensing plugin is the receiver: its matching validator accepts the same optional shape, migration 7 stores the latest receipt's snapshot on its occurrence bucket, and the admin detail API exposes it as hostResources. Replays cannot replace it; a later receipt without a snapshot clears it. The detail uses the host's published ResourceMeters for CPU, RAM and disk percentages and absolute IO/network rates. Historical occurrences have no extra section. Update the licensing receiver before enabling a core that emits the new field, since an older receiver's strict entry validator refuses unknown fields. The receiver needs no new host API; the existing UI API 55 floor remains sufficient for this field. As with every external shared consumer, the receiver must migrate to the shared API of the paired core (PLUGIN_SHARED_API_VERSION in packages/plugin-shared/index.mjs, currently 14) before the coordinated host update; its current capable core floor is 0.29.99.

Elowen is a TypeScript daemon with a Next.js web application. The daemon owns the brain, persistence, plugin registry, platform adapters, and authorization decisions. The web process presents those capabilities and proxies browser requests; it is not a second brain.

The live-emitter catalog PROBLEM_CODES is separate from the open report wire grammar. Producers use it through typed logger metadata, for example logger.warn('Retry exhausted', undefined, { code: 'provider.retry_exhausted' }). When reporting is enabled, ProblemService.observe validates new warning/error metadata against this catalog before aggregation; missing or undeclared codes become runtime.reporting_invalid or plugin.reporting_invalid without a text sample. Reporting disabled or local-only metadata collects nothing. This bounded validation uses the existing catalog and introduces no additional storage or transport. provider.search_replay_refused and alert.render_failed are retired only from live classification. Persisted batches still validate against PROBLEM_CODE_RE and replay their original codes and samples; licensing retains their historical cs/sk/en descriptions. The disputed subagent.continuation_failed entry and the skills.* vocabulary remain declared. A real consumer is ProblemService.project in src/problems/service.ts.