Navigace
ELOWEN · Dokumentace

Poslední aktualizace: 20. října 2026

Procházet dokumentaci · Voice and account surfaces

Tento článek zatím není přeložen. Je dostupný v angličtině.

Developer reference

Voice and account surfaces

Owning code

  • Settings voice section: web/modules/settings/VoiceSettingsSection.tsx.
  • Account voice section and the Users permission toggle: web/modules/account/VoiceAccountSection.tsx and web/modules/users/VoicePermission.tsx.
  • Call, ring and dictation state: web/modules/voice/ (callController.tsx, useVoiceCall.tsx, useDictation.ts, dictationSession.ts, socket.ts, IncomingCallPrompt.tsx, CallStatus.tsx).
  • Browser capture and playback: packages/plugin-shared/voiceAudio.mjs, imported as elowen-plugin-shared/voiceAudio; canonical sample rate and pcm16HasSignal live in voicePcm.mjs. Voice protocol source/mirror re-export the sample rate.
  • Composer entry points: web/modules/advisor/ComposerActions.tsx, web/modules/advisor/ChatComposer.tsx and web/modules/dashboard/HomeComposer.tsx.
  • Service worker notification handling: web/public/sw.js.
  • Daemon routes: src/api/routes/voiceCalls.ts, src/api/routes/voiceRings.ts and src/api/routes/selfSettings.ts. The ticketed /ws/voice/<id> attach lives in src/api/voiceWebSocket.ts, browser-credential checks in src/api/browserTransport.ts. Shared budgets and bounds: src/shared/voice.ts, which the browser mirrors in web/lib/voice.ts.

socket.ts owns clearVoiceSocketHandlers and consumeVoiceTicket(factory, sessionId, ticket, control) for calls and dictation. A cancelled reservation opens one socket through the injected factory, sends hangup or cancel after open, then clears handlers and closes. Error, close or a 15-second unopened timeout retires that socket without sending. This short cleanup belongs to the reservation independently of disposal or a new start, so a late response still frees the daemon slot before closing. Current-session audio, generation fencing and capture remain owned by their controllers; there is no retry or additional request.

Browser-login media carries credential provenance instead of a guessed origin. POST /voice/calls and POST /voice/dictations take browserTransportAuthority(c) (src/api/browserTransport.ts, full and impersonation scopes only), revalidate it through browserTransportValid before the paid start and answer 401 unauthorized when it lapsed; the engine starts inside webSocketTickets.runWithBrowserAuthority, so the call or dictation keeps that authority. When account security revokes an account's sessions it publishes auth-security (src/api/context.ts), and the subscriber in src/api/pluginWebSocket.ts drops the account's tickets and calls revokeBrowserSessions on both engines: a browser-login call or dictation ends as disconnected through the ordinary teardown and keeps its persisted dialogue, while plugin dictations, third-party calls and admitted model work continue. A start whose authority is no longer current fails with BrowserSessionRevokedError instead of a provider fault.

Gesture preparation and shared capture

Core calls and live dictation use createVoiceAudio and createVoiceCapture from elowen-plugin-shared/voiceAudio. Chatbot's hold control calls prepareVoiceCapture(capture, failed, signal) synchronously in its pointer/keyboard gesture. The returned start(): Promise<Pick<VoiceAudio, 'stop'>> alone requests getUserMedia after a confirmed hold. A short press calls stop without microphone permission. Stop and AbortSignal close the context, stop every track and fence late permission/worklet completion. One start per preparation; stopped starts reject AbortError. Capture resamples actual context rate to mono PCM16 24 kHz; Chatbot streams frames after ready and owns transport/draft state. Its existing live dictation engine sends partials while held and final text after release; maxCallSeconds from ready bounds capture, and no batch accumulation or automatic draft send occurs. The helper starts no work on import and owns no draft, upload or accounting. The Next workspace root includes packages/ and aliases the two browser-safe runtime entries, while TypeScript resolves their adjacent declarations. Next emits standalone/web plus workspace node_modules; scripts/build-web-bundle.mjs copies the web subtree and merges workspace dependencies into the existing flat web-dist layout, retaining server.js, .next/static and public for every launcher. Its staged publication retains the old artifact on build, entry or asset failures. No daemon module is bundled into the web. Secure context, microphone iframe delegation and Blob worklet CSP permission are required; see the shared package README. Actual Safari/iOS gesture behavior requires platform verification.

Voice and account spending

Settings/Voice configures the provider, models, bounded call/ring durations and the hold limit (voice.holdSeconds, 0 to 300 s in steps of 5, 0 reads "No holding") through the shared config mutation. Account/Voice edits personal voice, language, callbacks and the dictation pause through useSaveMyVoicePreferences; it reads the cached revision at each save, adopts acknowledged revisions and offers AutoSaveStatus's reload action on conflict. Language and callbacks remain editable when no voices are available. Config voice/provider saves invalidate the voice catalog (web/lib/mutations/system.ts). The personal picker uses catalog.active.voices, computed by the daemon, and ChoiceField pins the saved value. All cs/sk/en labels and hints live in the voice namespace. Both sections register stable row ids in rowAnchors, shared with siteSearch, so palette links reveal the actual control.

SliderWithReading composes the scalar shadcn Slider with a visible reading and aria-valuetext. Pass the usual Slider props and reading; readingWidth (a fixed w-10 or w-20 column, or auto) and action support a compact reading with a trailing reset button. Voice durations, Terminal and Account appearance share it; without an action only the track and reading render. It adds no requests or timers. SpendLimit reads MICROUSD_PER_USD from lib/format and accepts only SPEND_LIMIT_MICROUSD values (src/shared/spendLimit.ts, mirrored in web/lib/spendLimit.ts), the same bounds the store and the API schemas use. Its ladder is whole USD, its stops keep a stored custom limit and are filtered by isSpendLimitMicrousd, and the spend and the slider reading go through formatCost with two decimals, or none for a whole-dollar limit. Shell mounts VoiceCallProvider once inside BrainChatProvider. The composer's single Options menu (ComposerActions, trigger chat-actions, menu chat-actions-menu) sits left of send/stop in the chat composer and holds the work-mode choice and the voice actions. Its trigger shows one general sliders glyph, tinted outside Build, with the mode in its title and aria-label; the visible "Options" word joins it from md: up. The menu has a "Work mode" radio group (the daemon's kind:'mode' commands, run through runSlash) and, only when the account has the voice grant and the instance enables voice, a "Voice" group with "Call" and "Dictate". While a call is active the textarea and attachment/menu controls yield to a single-line CallStatus inside the composer: phone, tabular elapsed time and translated state. The primary action is a shared danger Button with PhoneOff, disabled during ending. The Options menu passes the composer's callActionRef to its close-focus handler, so selecting Call moves keyboard focus to hang-up after the menu releases focus. If a turn runs, the separate existing accent square STOP remains immediately before hang-up and only aborts that turn. The draft remains owned by BrainChatProvider. After ending, the textarea is editable immediately and a normal end dismisses the call state at once. Errors remain as dismissible alerts until acknowledged or a new call starts. A ready incoming ring explains unavailable acceptance in the body while keeping Decline available. The persistent controller exposes the snapshot, hangup and dismiss through useVoiceCallController, so compact and full composers share one call with no extra transport. Dashboard HomeComposer uses its startNew() action from a shared ghost/icon Phone button immediately before Send, with the same instance-enabled/account-granted availability and existing cs/sk/en call label. VoiceCallSession.start accepts either an existing session id or an async target factory receiving the preparation's AbortSignal: capture/playback still starts in the click stack, then the factory resolves before the paid call request. startNew(target?) creates the conversation through BrainChatProvider.startNewConversation({ target }) in the destination chosen in the dashboard, never opens the chat dock, and navigates to /chat while the persistent call connects. Ordinary chat Call and incoming rings still pass their existing ids. HomeComposer shows the shared CallStatus and hang-up while pending, disables sending during a call, and retains microphone/creation errors inline. Cancelling while creation is pending prevents a paid call and late navigation. Without voice availability the phone is absent; this adds no transport, polling, or provider request beyond the selected call. Call and dictation capture share one microphone request that asks the browser for echo cancellation, noise suppression, automatic gain control and voice isolation; GPT-Live has no noise setting of its own, so the browser is the only filter before the model and prompts/voice/call.md tells it to treat background sound as not addressed to it. Dictation is live: ChatComposer owns useDictation, which runs a DictationSession (microphone first through createVoiceCapture, the call's capture worklet without playback, then POST /voice/dictations and a same-origin /ws/voice/<dictationId>?ticket= socket) and writes the daemon's interim text into the textarea at the caret where dictation started while the person speaks; a final text replaces the item's interim text, and items are separated by single spaces. DictationDraft in web/modules/voice/dictationDraft.ts is the pure text model imported by useDictation; it owns an exact range, including inserted spacing, against its previous draft snapshot. The composer's native beforeinput listener calls useDictation.beforeEdit(start, end) with the selection before the change, and every textarea onChange calls useDictation.edit(value, caret). This anchors edits even beside identical text: edits before the range shift it, edits after it leave it intact, and intersecting edits detach its existing item IDs. Delta, final and withdrawal also reconcile the current snapshot; ownership never comes from searching for matching words elsewhere. Detached IDs' late updates are ignored. New item IDs insert at the current caret; cancelling removes only the new owned span and preserves manual edits. The pill in the trigger's slot (dot and clock, the clock hidden below md:, tick, cancel) stays compact so a 320 px composer remains readable; the send button stays visible and enabled beside it. Send (button or Enter) while dictating stops the dictation and waits for the daemon's ended before sending the composed draft. The shared voice contract's VOICE_DICTATION_STOP_BUDGET grants three seconds for committed finals and two seconds for upstream closure and usage settlement; the browser's six-second fallback includes a one-second transport margin and keeps receiving text throughout that wait. The tick stops and keeps the text; the cross or Escape cancels and removes only the text this dictation inserted. Holding the send button for 400 ms (HOLD_TO_DICTATE_MS) starts the same dictation for as long as it is held and releasing it is the tick: the text stays and nothing is sent, and the click that ends a hold is swallowed. It applies only when voice is available and no call or dictation runs; a short press still sends. While voice is available an empty-draft send button is aria-disabled and dimmed rather than disabled (a disabled button gets no pointer events), its click is ignored, and it carries touch-none select-none with the context menu suppressed so a touch long-press neither selects nor opens a menu. Keyboard is unchanged (Enter sends). 700 ms of silence after speech finishes a segment; the account's dictationPauseSeconds (Account → Voice: 3, 5, 10 or 20 seconds, or "Never"; default 5; null is "Never", VOICE_DICTATION_PAUSE_SECONDS in src/shared/voice.ts, offered by VoiceAccountSection.tsx) of continuous silence stops it. While connecting, the existing three-second pre-roll queue retains audio and commit boundaries in capture order; trimming drops a boundary only after all preceding audio for that segment has gone. Opening flushes the queue before any new controls. The microphone is released on every exit, and failures show translated toasts without touching the draft. An account already dictating in another tab receives HTTP 409 dictation_active; the toast asks the user to end that existing dictation instead of reporting a lost connection. Calls and dictation both use createVoiceSocket in web/modules/voice/socket.ts: it encodes the session ID, selects WS/WSS from the page origin and attaches the one-shot ticket to /ws/voice/<id>. The hooks provide it directly to their session dependency objects. useVoiceCall owns the call state machine: POST /voice/calls through elowenClient, then a same-origin /ws/voice/?ticket= socket. AudioWorklets capture mono PCM16 at 24 kHz, bound outgoing frames, play ordered daemon PCM and report consumed sample positions. After an empty queue, playback waits for 500 ms of PCM or at most 500 ms of audio-context time from the first queued frame, so irregular arrivals have a reserve and short final tails still drain without an item-end marker. Live streams output continuously, so the reserve is also how long Elowen keeps talking after the caller cuts in; measured provider gaps of up to ~300 ms ran a 150 ms reserve dry mid-speech. Waiting does not consume samples, advance played.ms or show speaking. A 5 ms gain ramp smooths the start, queue exhaustion and recovery when a frame arrives during fade-out; queued samples are all consumed and the final position still reports speaking=false. This uses the existing VoicePlayback worklet and played callback, adds no wire controls, and leaves microphone capture unchanged. One audio_started binds the continuous Live stream to itemId; caller transcript fragments leave playback running while Live controls speech. Cancellation closes audio contexts, tracks, worklet ports and sockets, including a microphone permission arriving after cancellation. Calls never contact a provider from the browser. Hangup stops capture/playback immediately and retains the control socket until authoritative ended, using VOICE_CALL_STOP_BUDGET.sessionDrainMs plus its browser transport margin (VOICE_CALL_STOP_BUDGET in src/shared/voice.ts, applied in useVoiceCall.tsx). The daemon's own live shutdown bounds the same drain with shutdownDrainMs (3 s) in src/voice/live/session.ts, so it settles inside the five-second pause checkpoint guard (PAUSE_CHECKPOINT_GUARD_MS in src/daemon/shutdown.ts). Sticky submitted admission frames report accepted work independently of later transport loss. Ending never restores transcript text into the composer, switches the selected conversation, or opens a recovery dialog. The daemon persists the unsubmitted dialogue through its non-triggering brain append seam, including a tail after accepted work. Its post-acceptance resync refreshes the existing chat snapshot rather than announcing a new user turn or synthesizing idle: the call card appears without a false thinking state, and an existing streamed reply stays intact. While main-agent work is still running, the same atomic snapshot carries accepted pending input as conversationAppends, separate from its settled history and live replay. The transcript projection places these call cards after the replay, while subsequent text and tool updates continue before them. A safe journal flush retires the pending projection and a new snapshot keeps the one committed card in history; this display never inserts input between a native tool call and its result. The saved call may have an empty display label; its localized header and expandable transcript remain visible. Without a saved dialogue there is no new card, and an unsent typed draft is left unchanged. The daemon's state frames are connecting, listening, on_hold and ending; actual worklet playback adds the local speaking indicator while listening. Live also streams exact-zero PCM during silence, so queue occupancy does not imply speech. The shared pcm16HasSignal helper (packages/plugin-shared/voicePcm.mjs, imported directly by browser audio and the Live engine; protocol constants retain byte-identical source/browser mirrors because the web root cannot import outside itself) marks each queued frame; the worklet reports whether the frame being played has signal and clears the indicator on silence or an empty queue. The internal played callback carries speaking, while wire played.ms still counts all samples continuously. No samples are dropped, no caller VAD is added, and dictation has no playback callback. The elapsed clock stays outside the live region so it does not interrupt announcements every second. on_hold shows "Elowen is working" in the composer's status line, with hang-up unchanged and no tone (the daemon holds the line while the work it started runs, see "Holding the line, and calling back" in docs/ARCHITECTURE.md). During that wait, Elowen can speak up to three short progress updates, at least twenty seconds apart and the first one twenty seconds after the work starts (PROGRESS_THROTTLE_MS and PROGRESS_MAX_CHARS in src/voice/live/session.ts). Each comes from assistant text that a following tool or step event closes, not from every written message. She is instructed to wait while the caller speaks. Updates keep on_hold active; the final answer returns the call to listening as before. The persistent controller validates /chat?voiceRing= with elowenClient.voiceRing and subscribes to private voice_ring / voice_ring_end payloads through voiceEvents on the shell-owned useElowenEvents core SSE stream. The controller opens no connection. Hidden-tab presence, reconnect and revive remain owned by that shared stream. pluginEvents and voiceEvents both use createModuleEmitter: subscribe returns an unsubscribe function, emit synchronously fans out a payload and isolates subscriber errors. With no subscribers there is no work beyond parsing, no buffering and no extra transport. The shell parses ring JSON; the controller validates payloads (including kind: question or done) and renders malformed-ring errors; a done ring takes the title "Elowen has the result". IncomingCallPrompt uses the shared Modal and overlayStack with no feature-local portal. Acceptance always requires an in-page tap, which activates microphone/playback before asynchronous navigation and starts the call with ringId. Decline uses elowenClient.declineVoiceRing. Missing or expired declines become stale; transport failures retain an explicit retry. Settlement, expiry and account unavailability disable acceptance. Call refusals show their specific translated code. Visible call state follows daemon echoes; useNow supplies elapsed time. A call requests microphone and playback in the click stack. When the Permissions API already reports the microphone as granted, it requests the daemon reservation at the same time, so the provider session opens while the browser prepares audio; otherwise it waits for capture first, so a denied microphone never makes a paid request. The call socket opens once both succeed. When either fails or the attempt is cancelled, the other is undone: prepared audio stops, and a reservation the attempt no longer uses is consumed and hung up instead of waiting out its attach deadline. Cleanup also owns late-reservation sockets and their deadlines. The service worker handles ring/ring_end push variants, restricts ring navigation to the app origin, and opens the app when an authenticated decline fails. A ring notification vibrates in a call pattern where the device supports it; browsers give notifications no custom sound, so its sound stays the system default. While IncomingCallPrompt can be accepted, useRingtone (web/modules/voice/ringtone.ts) loops a synthesized ringtone through its own AudioContext and closes it on accept, decline, expiry or unmount. A page without earlier user interaction keeps that context suspended and stays silent.

The fake daemon's voice handlers script REST, private ring/core SSE and ticketed WebSocket audio. POST /__test/voice controls ring, ring_end, audio, emit, drop, admissionError, catalogMode, reset, callScript and dictationScript (the /__test/voice handler in web/tests/e2e/fake-daemon/handlers/voice.ts); GET returns call frame/control observations without tickets. Both the automatic POST /__test/reset and the voice reset control share one voice-state reset: close live transports and clear calls, dictations, rings, admission errors and scripts. Connecting-dictation assertions pin the admitted session ID so an earlier commit cannot satisfy the new session's wait. The audio script emits audio_started before PCM. The voice Playwright fixture forwards only /ws/voice/ to the local fake daemon because the test Next server has no nginx upstream. Product sockets remain same-origin. Playwright launches with fake microphone permissions and devices. voice.settings.e2e.ts covers instance settings, account choices, and the user voice grant in cs/sk/en at 320 px with a mobile context and 1440 px, including loading, empty and recoverable catalog errors and the per-second call-price help. voice.call.e2e.ts covers full-duplex playback, hold, goodbye, failures and persisted call cards without composer restoration in cs/sk/en at both widths; caller transcript deltas do not flush playback. In-process route tests share createVoiceSpendApp; the fake user PATCH uses the real userPermissionsSchema. Browser verification belongs to integration.

Browser call fixtures seed streaming and programStatus through the current control contract when asserting Stop; the retired running field does not establish an active turn.

Voice preferences and call transcripts

Settings → Voice mode uses the existing config query and config mutation. The provider and four model pickers (call, transcription, dictation and speech) consume GET /voice/catalog; only GPT-Live 1 is offered for calls, with shared help explaining USD 0.05 per minute billed per second, paid waiting and separately billed work. More history does not change that call rate. No provider eligibility or model/voice list is inferred in the browser. Call and ring sliders mirror the shared protocol bounds using literal contract types. An empty provider catalog prevents enabling voice but leaves disabling and duration edits reachable. Read failures replace controls with the shared retry state; invalid enabled configuration cannot save.

Account → Voice uses the shared voice-preferences query in modules/voice/voicePreferences.ts. Catalog and revisioned account responses are validated at the HTTP boundary. Autosave patches only changed preferences through PATCH /auth/me/voice with the last acknowledged expectedRevision. A conflict remains an explicit failed save with the draft retained. Reloading the saved preferences is an explicit action that discards the draft; there is no automatic overwrite. The catalog's active selection identifies the configured call and speech models for every eligible account without granting access to instance config. The voice picker offers the intersection of those models' voice lists, or the single configured model's list when only one is set. Inactive models never restrict the selection. A null active selection, missing provider/model or empty intersection makes voice selection unavailable while language and callback preferences remain editable. A previously saved unsupported voice stays visible as the current value but is not a selectable option. The Users pane independently autosaves only can_use_voice, including for administrator accounts. The voice dropdown is built by useVoiceOptions in web/modules/voice/voiceSamples.tsx: each GPT-Live voice shows a presentation and accent from a fixed trait table, and a sample button plays its recorded Czech clip from web/public/voices/<voice>.m4a. One clip plays at a time. Realtime voices have neither description nor sample. web/tests/modules/voiceSampleParity.test.tsx keeps the table, the clips and the daemon's LIVE_VOICES in step.

Both user-bubble layouts render a turn with typed voiceCall metadata through the shared VoiceCallMessage (web/modules/voice/VoiceCallTranscript.tsx) in place of the plain text: a phone header with the call duration, the optional message text (an admitted request's interpreter summary), and that message's own transcript segment collapsed under it as plain-text speaker entries. A non-triggering tail has no inferred summary. History and live-user projections retain the same record. The local VoiceTranscriptEntries renders the plain-text speaker list for this disclosure, preserving line breaks and using the shared empty state when there are no entries. Rendering is linear in the supplied transcript and performs no reads. Brief headings are never parsed to recover a transcript.