Shared desktop UI
Shared desktop UI
DesktopPreview imports useDesktopActionMarker(action, actionText) and useDesktopActionRing(action)
from web/components/desktop/useDesktopActionMarker.ts. This internal module owns action identity,
receipt-based expiry, the newest pending replacement and the still's click-ring timer; it adds no DOM
or polling. Missing action or text clears the marker, while text without an action remains the caller's
untimed presentation. Started actions have no expiry timer; finished markers, pending replacement and
rings each need at most one timer, all cleared on unmount. The timing and failure rules below apply to
both Browser and Sandbox through their shared DesktopPreview consumer.
The internal useVncSurface hook uses the sole noVNC declaration in novnc.d.ts and returns only
{ state, aspect }. Its private container is appended to the supplied slot and moved when the slot
changes, keeping the same client and canvas. An actual dynamic-import failure is logged and reported
as failed without scheduling a reconnect; the typed default export needs no optional-export fallback.
Neither this viewer hook nor the action hooks are published on the plugin UI runtime.
UI API 39 adds shared narration and pending-question presentation to API 38's viewport-gated still polling and API 37's all-width collapsed stills and control-stream suspension: components.DesktopPreview and hooks.useDesktopPreviewState(), beside the control client of API 35 (hooks.useDesktopControl(plugin, sessionId, { controlsEnabled, streamEnabled })). The viewer hook useVncSurface is not published (see above). Contracts live only in packages/plugin-ui-kit/index.d.ts; the component is web/components/desktop/DesktopPreview.tsx and its sheet desktop.css. Browser's and Sandbox's chat dock views, Sandbox's Project panel and Account Desktops section are its consumers, so a browser session and a managed Project look and behave identically: a small computer tile that enlarges. UI API 41 adds the optional caption: what sits under the collapsed tile beside the claim control, in place of identity, while the expanded controls row keeps identity. Absent, the collapsed tile shows identity. Sandbox's chat dock view passes its Project's compact ResourceMeters there, so the chat tile shows live CPU, RAM and disk use instead of the Project name; Browser passes none.
Minimal use, in a producer that already knows its desktop session id:
const control = hooks.useDesktopControl('sandbox', sessionId, { controlsEnabled });
const preview = hooks.useDesktopPreviewState();
return <components.DesktopPreview control={control} preview={preview} title={s.desktopTitle} placement="chat" controlsEnabled={controlsEnabled} narration={narration} pendingInput={pendingInput} />;
The producer keeps its own useDesktopControl client and supplies identity, actionText, navigation and power. The host owns narration and pendingInput, used identically by Browser and Sandbox. Narration is visible only in the expanded view, hides after ten seconds or dismissal, and stays hidden while that reply streams; an empty narration resets it for the next reply. Text clamps to three lines, two on phones, with padding outside the clamped inner box and a click-through surface except for dismissal. A persistent polite live region announces the pending question. Its button closes the overlay, waits for focus restoration through the existing zero-delay timer, then calls pendingInput.reveal. No content means no notice space. There is no notices render slot or slot context in API 39; producers must pass these typed props instead. power.dialog mounts the producer's confirmation inside the expanded surface, sharing the overlay depth. Closing the view never stops the desktop. The expanded controls row carries no name at any width: identity (or caption) labels only the collapsed tile, and title names the expanded dialog for assistive technology. Notice content scrolls in a bounded area above controls without changing the framebuffer aspect ratio.
useDesktopPreviewState() is the shared layout decision (at or above 768px, the complement of the phone breakpoint) plus the expanded flag, returning { expanded, setExpanded, wide, liveView }. liveView is the default framebuffer policy wide || expanded for a tile without a still. A supplied still overrides collapsed live rendering at every width unless this view holds a lease, so its producer polls while !preview.expanded && !control.lease; there is no second matchMedia in a producer. DesktopStillPreview is Browser's existing contract: { dataUrl: string | null, aspect: number | null, state: 'loading' | 'ready' | 'stalled' }. Browser honors the thumbnail response's refreshMs and liveForMs, pauses hidden-document polling and never shows an expired picture. Every Sandbox tile uses this same contract through its shared DesktopTile. The producer supplies its capture function and account/session query key to hooks.useDesktopStill({ queryKey, enabled, queryFn }). The shared host hook validates bounded inline PNG/JPEG/WebP responses, follows server cadence, pauses hidden-document and off-viewport reads, and clears failed or expired images. Pausing while the tile is expanded is the producer's job through the enabled option; the Sandbox adapter passes !!accountId && !expanded. Its DesktopStillHandle.ref is attached by DesktopPreview to the tile root through IntersectionObserver; standalone consumers must attach it themselves. The Sandbox adapter calls the authenticated plugin route /plugins/sandbox/api/desktop/still?projectId=ID (declared as desktop/still with access: 'user') and keys by account, Project, generation and epoch. A Sandbox tile that may take control (controlsEnabled, the chat dock tile) keeps control SSE open while collapsed, so an agent's request for a person reaches it; a watch-only tile uses streamEnabled: preview.expanded to open control SSE only in the raised view. While the tile holds a lease its still pauses, because the held lease keeps the collapsed tile live. A suspended tile can neither see control change nor be driven, so suspension gives an owned lease (or an owned pending handover) back through an explicit release and forgets the control snapshot rather than show a stale holder; the next expansion receives a fresh one. A release the server answers with 409 (the lease already ended) counts as done, not as an error, and a heartbeat answer for a lease that is no longer current is ignored. A new session resets the snapshot; a ready still with suspended SSE does not report Connecting. Toggling this option preserves the mounted view identity; it defaults to true for producers that need collapsed session updates.
Behavior the component owns: one useVncSurface canvas per preview, never a second thumbnail connection that pays the producer's full-framebuffer encoding cost again. With a supplied still and no held lease, a collapsed tile has no ticket, socket or canvas, and its only claim affordance is the Take control described below that opens the screen; opening dials the one framebuffer, and closing disconnects it and reports the view as not connected. Without a still, a wide tile stays live and reparents its existing canvas on expansion, while a phone tile shows the instruction to open the screen and dials only while expanded. A closed stream renders visible Closed in both canvas variants, never a connecting spinner or an instruction to open. Remote input follows mobileInput (default false, so a phone viewer is watch-only; Sandbox opts in, Browser does not) and a collapsed tile never forwards input. The component also owns status mapping (closed, viewer limit, disconnected, stalled still, handover pending, who controls), the connecting, failed and unavailable states, the claim and release control with pending and error feedback (a failed claim is a toast, never silent), and the report of the connected view, which the automatic handover waits for. A claim is offered only without a local lease and with a live framebuffer view, disabled until it is connected. A collapsed still or phone tile claims nothing itself; while the agent asks for a person it offers a pulsing Take control that only opens the screen, where the connected view takes the requested control. A lease this view holds keeps its framebuffer connected even when collapsed, because the claim lives only as long as that socket, so a tile collapsed while in control stays live and keeps offering hand-back, which pulses while the agent waits for it. An owned pending handover offers only hand-back. Only the collapsed canvas slot is pointer-transparent, so clicking anywhere on its picture opens the overlay even when noVNC consumes canvas events; the expanded full-framebuffer surface keeps remote pointer input. With controlsEnabled: false no claim is offered and the hook refuses any. Escape in a view-only surface closes it; Escape typed into a claimed canvas stays with the remote machine because useVncSurface marks that container data-keyboard-claimed, which the shared dialog honors. The same mark decides the pointer: core forwards Cursor -239 and VMware alpha cursor 0x574d5664 only on a lease-bearing connection. A controlling viewer gets the real remote sprite as noVNC's local CSS cursor and frames without an embedded pointer, so its position follows the local mouse immediately. Watch-only connections get the remote pointer drawn into their picture; noVNC then hides the local pointer, so desktop.css restores the ordinary pointer on every surface that is not claimed. Claim and release reconnect with the corresponding ticket and cursor negotiation; the existing CSS rule remains unchanged. All copy comes from the core desktop dictionary (cs, sk, en); a producer sends only its own words (the identity, the action text, the stop confirmation).
Live activity: stream.action (DesktopAction in the kit) carries { kind, target?, id?, status?, label?, text?, key?, direction?, point?, step?, message?, receivedAt }. Every field after kind is optional, so Browser's legacy { action, target } event keeps parsing; point is a fraction 0..1 of the framebuffer, text a short preview that already ends with … when the daemon cut it, label the target as written on started and the element's plain display name on done or failed (a bare reference such as e123 names nothing a reader can use, so the producer shows no name for it), and receivedAt the host clock at arrival. The producer composes actionText from it; the component owns how long that line is visible, so Browser and Sandbox behave alike and no prop was added. A started action stays until its own done or failed arrives (same id), a done stays two seconds and a failed five, counted from receivedAt (a re-render, a late mount or a new text for the same event never restarts the time), then the line clears. An event without status counts as done at receivedAt, and one already older than its lifetime when the tile mounts is not shown. A newer action replaces the shown one, but every shown action stays up at least 400 ms (the newest waiting action wins); the same operation's done updates its line at once. The stream merges that done or failed into the operation's started event (same id), keeping the label, text, key, direction, point and step it did not repeat, so the finished line still says what was typed or pressed. The same line is drawn beside the state dot as one truncated line, on the collapsed tile and on the expanded surface, with the same timing; while the surface is open the tile behind it shows only the dot, so the line is not drawn twice. One polite live region announces an action only once it is done or failed, never started: it sits beside the tile button while the tile is collapsed and moves into the dialog next to the pending-question region while the surface is open, so there is never more than one. With no stream.action the producer's actionText is shown as given.
Click ring: when the collapsed tile shows the still (not the live view), the first event of an action that carries a point draws a ring there for about 480 ms, like the one the guest composites into live frames. It is an aria-hidden element positioned in percent of the still's own box (.desktop-preview__still-frame, the picture's shape fitted into the tile, so letterboxing does not move it), once per action, never for a failed one (and a failed event takes away the ring its own started drew), and under prefers-reduced-motion a static dot for the same time. The live view draws no ring in the web: the guest already composites its own into live viewer frames, so each surface has exactly one mechanism and the framebuffer is never ringed twice. Screenshots the agent takes stay free of it.
placement="chat" marks the tile root with data-chat-dock. On the full chat page at 768px and above desktop.css positions such a tile bottom-right above the composer. BrainChatSurface alone measures floating, non-expanded tiles, stacks them bottom-up with --chat-dock-offset, publishes the aggregate --chat-dock-height, union --chat-dock-width and --chat-composer-height. Browser and the managed Project desktop use this same contract; a producer supplies only placement="chat", never clearance or offsets.
UI API 42 adds the chat dock slot (web.chatDock, registration chatDock). BrainChatSurface mounts PluginChatDockHost (web/lib/pluginChatUi.tsx) once, after the loaded turns, and hands each contributing bundle { plugin, project, sessionId, narration, pendingInput }, where project is the conversation's projectRef from the status. The view is drawn from live state, so it does not depend on which history page is loaded: Sandbox draws the running desktop of the conversation's managed Project there and Browser the live sessions this conversation opened, so a reload, which loads only the newest history page, can no longer leave a tile in an unloaded turn. A daemon restart keeps the Sandbox tile, because the Project desktop keeps running, but ends the Browser one, because Browser closes its sessions on stop and at boot. Without a contribution nothing is drawn; a view that throws shows the host crash notice in its place only. The surface hands the dock the narration and the pending-input notice directly; no transcript row reads either, so a streamed token never re-renders a settled turn. UI API 43 removes the anchored chatArtifacts registration the dock replaced.
The full desktop transcript is one flow-root containing two React-rendered right floats: a zero-width pusher of --chat-dock-float-top height, then a clear:right exclusion of tile width plus gap and --chat-dock-float-height. The single geometry effect moves these adjacent sibling nodes, without replacing them or changing their parent, before the last in-flow block whose start is at or above the unwrapped end minus the exclusion height. Both float variables live on their respective float elements, not on the surface; pusher height is relative to that block's measured off-float origin. Probes therefore invalidate the intersecting suffix rather than all preceding history, while the same BFC still owns every turn. Turns, bodies, segments and log columns stay ordinary block flow, so lines above the exclusion retain full width and all following prose wraps beside it across turn boundaries. Leaf tools, cards, code and media shrink as whole boxes. Padding preserves the former independent flex/grid spacing. During active exclusion only, data-chat-dock-tail on the actual final segment trims trailing segment padding and the last markdown margin when no metadata, pending input or ambient card follows. Marking that segment rather than an ancestor avoids invalidating inherited styles across the entire transcript. These are separators between content, not content at the end sentinel; retaining them adds a blank band on top of the discrete line-wrap step. The single geometry effect owns the marker, and above-zone, absent-dock, phone and compact layouts retain their spacing. Only the marker slot is positioned: positioning a turn would incorrectly re-anchor its nested absolute desktop tile. The BFC contains the float and extends scrolling itself, without bottom-clearance padding.
The single geometry effect reads the last in-flow end sentinel with both floats off. Eligibility is read with both the floats and the tail-spacing marker off. When eligible, a second off-float read supplies the trimmed active tail's search bound in the same frame. Only content whose original unwrapped end, scrolled to the bottom, reaches the tiles gets an exclusion; that decision cannot oscillate with float-induced growth. The pure pixel search finds the first top where the measured tail fits in the stack height plus tile gap. It warm-starts from the previous top with at most two predicate checks, then bisects the unwrapped-end interval in at most 24 probes. Each probe forces synchronous browser layout inside the same animation frame; it changes only local float heights, while final geometry is published only when changed. An unchanged resize notification first checks the current sentinel and the solved width, stack dimensions, viewport height and composer position, then reuses the solution without writes or probes. Text, child-list, details, media, fonts and viewport-inset changes invalidate this proof even if the full-width end stays equal; the effect ignores only its own float relocation records. No token is throttled or deferred. Probe count alone is not a performance guarantee: actual suffix layout cost must still be measured with streaming browser fixtures. No geometry is inferred from the last prose block or from ink heuristics. Expanding/removing tiles releases their share; absent floating tiles there are no float boxes or dock variables, and content ending above the dock has no float top/height. Phones and the compact dock retain inline tiles and their existing flow. placement="inline" is always in flow, bounded to the tile width, as in the Sandbox Project panel. Absent a producer there is no tile; absent a slot there is no such control.
Browser verification lives in chat.dock-float.e2e.ts and chat.project-desktop.e2e.ts, using the real Browser and Sandbox bundles. The matrix takes its skin IDs from the production catalogue, including the current DEFAULT_SKIN, rather than treating the removed default ID as a design. Replacing settled fixture history after streaming requires a terminal idle frame: a snapshot deliberately replays the session's unsettled journal across navigation. The 300-turn streaming fixture loads every page through reader scroll gestures, deriving its page count from CHAT_HISTORY_PAGE_SIZE, then runs the same 60 delivered chunks both without tiles and with two tiles. chatFrameCost.ts records renderer CPU and layout counters per chunk, separately times every animation-frame callback measuring the shared tile roots, and counts end-sentinel reads. The historical implementation can use the identical recorder without an end sentinel. The spec enforces median total geometry per chunk and tile CPU overhead over the paired no-tile baseline below 10ms, giving the 8ms target 2ms of host-noise headroom, and geometry callback p95 below 12ms. Summing callbacks per chunk prevents cheap follow-up resizes from diluting the expensive work. Screenshots capture both settled tool/task tails and the subsequent streaming reply; geometry assertions still reject overlaps, missing full-width lines and a tail-to-dock gap exceeding one line. Full-width proof is collected before the transcript viewport clips scrolled-away lines, while overlap checks use painted, clipped rects, including the padded code-block and user-message bubble boxes rather than only their glyphs. The one-line gap bound uses the text line at the exclusion boundary, where moving a pixel can add or remove that whole line, not a smaller trailing task caption. Additional live layout probes verify that the chosen pixel fits and its predecessor does not.
The expanded surface is Modal with chrome="bare": an aspect-fitted box on the soft scrim whose pointer events are restored explicitly (a bare surface is not a pointer target, so the space around the box is the scrim that closes the view). Overlay stack, inert isolation, the focus trap, return of focus to the tile and scroll lock are the shared ones; Sandbox's Project panel therefore keeps its drawer inert underneath while the desktop is raised.
Project-register web.projectRows contributions may supply UI API 48 nameIndicator, using the same localized label/icon/tone status shape. Core draws the first contributor's indicator beside each name with the shared HelpTip and accessible Project-qualified label; no contribution draws nothing. Sandbox derives it from the same bounded useDesktopOverview cache as Account and chat, intersected with running environment state. Resource RAM remains the whole environment's measured RAM; no separate desktop memory estimate is fabricated. Account uses the shared danger IconButton and ConfirmDialog, and Sandbox's useDesktopStop invalidates that overview immediately after the human-only plugin route settles. The enlarged desktop passes DesktopPreview.power, exactly like Browser: the existing shared More actions menu contains Quality then Speed, followed by the same danger power control. The same confirmation mounts inside the existing overlay stack; pending disables the control and errors use translated toasts.
The expanded DesktopPreview owns a More actions menu with Quality and Speed radio choices for both watch-only and controlling views. It stores the choice per browser through the shared usePersistentState slot elowen-desktop-picture-mode; the default is Quality. It passes the optional VncSurfaceProps.pictureMode: 'quality' | 'speed' to useVncSurface, without changing the public DesktopPreviewProps contract. A bare hook defaults to Quality when the prop is absent. The same connected noVNC client changes its quality/compression levels in place: Quality uses 8/6, Speed 2/1, and reconnects use the current choice. The guest recognizes level 2 with desktop-resize support as a request for a 75% framebuffer and q50 JPEG for large updates; other servers, including Browser's x11vnc, simply receive the lower JPEG quality. No producer-specific branch, second viewer, capture-resolution change or control-authority change is involved. A choice applies to that view immediately and is restored by later mounts; it does not force other already-mounted views to change. Copy uses the core desktop dictionary in cs, sk and en.
The host owns the single noVNC dependency, lease heartbeat, revision fencing, bounded event parsing and private ticket exchange. Because the viewer hook is not published, a producer cannot drive a bare viewer; both current consumers (Browser and Sandbox) use DesktopPreview, which passes the control client's ticket, connectionKey and interactive to the hook. Automatic agent-requested handover waits for a connected, controls-enabled view. useVncSurface reports connected only for the socket of its current enablement and connectionKey: while disabled, and on the very render that re-enables it or changes the key, the state is connecting, so opening a tile whose earlier socket had connected cannot claim before the new socket exists, which the server refuses with 409. Historical and read-only cards pass controlsEnabled: false: the shared hook refuses explicit and automatic claims, stops heartbeats and releases any claim if the flag changes, including a claim still awaiting its response. Claims reconnect with a fresh proof-bearing ticket, while pending control remains view-only. Lease tokens stay in the mounted client's memory; remounts never recover authority from sessionStorage. Reconnects use bounded backoff and a longer viewer-limit delay. Disabled and empty sessions do not connect. Server input checks remain authoritative regardless of noVNC's viewOnly flag.
Owning code
web/components/desktop/DesktopPreview.tsx: the tile and expanded surface, narration, pending-input notice, status mapping and claim controls.web/components/desktop/useVncSurface.tsandweb/components/desktop/novnc.d.ts: the single viewer connection, reconnect policy and picture-mode encoding.web/components/desktop/useDesktopControl.ts: lease claim, release, heartbeat and ticket exchange.web/components/desktop/useDesktopStill.ts,useDesktopPreviewState.tsanduseDesktopActionMarker.ts: still scheduling, layout state and action timing.web/components/desktop/desktop.css: tile, surface, narration and click-ring styles.web/lib/pluginUi.tsx: publishesDesktopPreview,useDesktopControl,useDesktopPreviewStateanduseDesktopStillon the plugin UI runtime.web/modules/advisor/BrainChatSurface.tsxandweb/lib/pluginChatUi.tsx: the chat dock host and its geometry.packages/plugin-ui-kit/index.d.ts: the public contracts (DesktopPreviewProps,DesktopStillOptions,DesktopAction).