mirror of
https://github.com/screenci/screenci.git
synced 2026-09-19 08:57:46 +08:00
0f211bea64
Accept codegen requests queued while no machine was connected: the poll loop applies them like any other edit and logs "queued by <name>" attribution. Update the editor docs to describe pending edits and code sync.
867 lines
41 KiB
Markdown
867 lines
41 KiB
Markdown
# Editor
|
|
|
|
Editor is the ScreenCI web app's editing surface for a video: a live preview of
|
|
the raw recording, a multi-track timeline, and panels for narration, overlays,
|
|
and render options. You edit visually in the browser, and every change is
|
|
written back into your `.screenci.ts` source through a connected
|
|
`screenci edit` machine, so code stays the single source of truth.
|
|
|
|
**You can edit without a connected machine.** Anyone in your org can open a
|
|
video and change narration, overlays, render options, and the rest right away.
|
|
Those edits render immediately in the preview and in exports. Because code stays
|
|
the single source of truth, each edit is also queued to be written into your
|
|
`.screenci.ts` source: the sidebar shows an "N edits pending" list ("not yet in
|
|
code"). Run `screenci edit` in your project and the next connect drains that
|
|
queue into the sources, attributing each edit to whoever made it. A teammate can
|
|
make the edits and a developer's machine can pick them up later.
|
|
|
|
Edits that change what is captured (record options, interaction timings,
|
|
on-screen text, the language set) cannot take effect until a recording runs.
|
|
They are queued the same way, but the editor badges them "applies after next
|
|
recording" and marks the preview stale rather than pretending they took effect.
|
|
Trigger a re-record (via CI, or a connected machine, which auto-records once it
|
|
applies the edit) to bake them in. While a connected machine is actively syncing
|
|
a video's source, that video's editing controls lock briefly until it finishes.
|
|
|
|
**Everything is editable by default.** Every feature a video declares
|
|
(narration, overlays, languages, render and record options) can
|
|
be edited in the web app. There is nothing to opt in to; the only choice you
|
|
make in code is where the content starts:
|
|
|
|
- **Arrays declare blank editor-owned names.** `video.narration(['intro'])`
|
|
keeps the name in code (so the body can call `narration.intro`) while the
|
|
content is filled in on the Editor page.
|
|
|
|
- **Plain objects are code values.** `video.narration({ intro: 'Welcome' })`
|
|
supplies the content from code. It is used at record time and stays fully
|
|
editable in the web app: once a value is edited in Editor, the Editor value
|
|
wins over the code value on every later upload.
|
|
|
|
- **Edits write back to code.** Whichever form declared a value, editing it in
|
|
the web app produces a code change applied by your connected `screenci edit`
|
|
machine, so the sources always show what the video renders with.
|
|
|
|
The `video.narration` and `video.overlays`
|
|
declarations type the matching fixtures to exactly those names, so a typo is a
|
|
compile error. The fixtures (`narration` and `overlays` in the
|
|
test body) expose the controllers regardless of which form declared
|
|
them.
|
|
|
|
The declaration forms at a glance:
|
|
|
|
```ts
|
|
import { video } from 'screenci'
|
|
|
|
// Blank editor-owned names: the names live in code, the content in Editor.
|
|
video.narration(['intro', 'outro'])
|
|
video.overlays(['intro', 'logo'])
|
|
|
|
// Plain objects: code values, used at record time, editable in the web app.
|
|
video.narration({ en: { intro: 'Welcome', outro: 'Thanks' } })
|
|
|
|
// Languages: the code set. Adding a language in the editor writes it here.
|
|
video.languages(['en', 'fi']) // the code language set
|
|
video.languages({ languages: ['en', 'fi'], mode: 'shared' }) // set with capture options
|
|
|
|
// Render / record options: code values are the starting point. Editing them in
|
|
// the editor writes the change back into these builder calls (renderOptions
|
|
// supports per-language overrides via { default, <lang> }).
|
|
video.renderOptions({ output: { aspectRatio: '9:16' } })
|
|
video.recordOptions({ fps: 30 })
|
|
```
|
|
|
|
#### You will learn
|
|
|
|
- [how the editor is laid out and what each part does](#the-editor-at-a-glance)
|
|
- [how pending edits sync to code](#pending-edits-and-code-sync)
|
|
- [how to edit and export a video in Editor](#editing-in-editor)
|
|
- [how to record from the editor](#recording-from-the-editor)
|
|
- [how to manage narration from Editor](#editor-narration-from-code)
|
|
- [how to use uploaded media as narration](#narration-media-from-editor)
|
|
- [how to manage overlays from Editor](#editor-overlays-from-code)
|
|
- [how render and record options combine with web edits](#editor-render-and-record-options)
|
|
- [how to manage languages from Editor](#editor-languages-from-code)
|
|
- [how to place effects from code](#effects-in-code-block-wrappers-and-gap-sleeps)
|
|
- [how web edits reach code](#how-edits-reach-code)
|
|
- [how action parameters are tracked and overridden](#action-parameter-tracking-and-overrides)
|
|
- [how to migrate from the removed `editable()` helper](#migrating-from-editable)
|
|
|
|
<!-- screenci-doc-video:docs/guides/editor -->
|
|
|
|
## The editor at a glance
|
|
|
|
Opening a video in the web app opens the editor. The page is laid out as:
|
|
|
|
- **Live preview** (center): plays the raw recording with your edits applied
|
|
on top, render-free. Camera zooms, cursor paths, overlays, narration audio,
|
|
and subtitles are all previewed live, so you see the result without spending
|
|
an export. A **Live preview** badge marks this mode. Some edits happen
|
|
directly on the video: drag an overlay to move it (a corner to resize),
|
|
pause and drag the cyan handles to reshape a cursor path, or type into the
|
|
subtitle box to change a cue's text.
|
|
- **Timeline** (bottom, resizable): rows for **Overlays**, **Zooms**,
|
|
**Interactions**, **Recording**, and **Narration**. Click or drag to seek,
|
|
scroll to pan, pinch or Ctrl+scroll to zoom (1x to 60x). Selecting a bar
|
|
opens its editor in the side panel; dragging a bar moves it, and dragging an
|
|
overlay's right edge changes its duration.
|
|
- **Side panel** (right): render options (canvas, background, roundness,
|
|
shadow, padding), recording options with a visual crop editor, and the
|
|
editor for whichever timeline item is selected.
|
|
- **Sidebar** (left): the language picker, the **Editor** view, the
|
|
**Exported** group listing every exported version, and the **Recording**
|
|
group showing your connected `screenci edit` machine and record actions.
|
|
- **Top right**: undo and redo (up to 20 steps, Cmd+Z / Shift+Cmd+Z), export
|
|
status, and the **Export** button.
|
|
|
|
## Pending edits and code sync
|
|
|
|
Every edit is ultimately a code change: code stays the single source of truth.
|
|
But you do not need a connected machine to edit. Anyone in the org can change a
|
|
video, and the edits render immediately while they wait to be written into the
|
|
`.screenci.ts` source. The sidebar's **pending** list shows how many edits are
|
|
"not yet in code" and who queued each one.
|
|
|
|
To flush the queue into your sources:
|
|
|
|
1. Create a personal editor token on the Secrets page and add it to your project
|
|
env file as `SCREENCI_EDIT_TOKEN=<token>`.
|
|
2. Run `screenci edit` in your project.
|
|
3. On connect, the CLI writes every queued edit into the source (logging
|
|
"queued by <name>" for edits a teammate made) and the pending list drains.
|
|
|
|
If an edit can no longer be applied (its target was renamed or removed in code,
|
|
or the source drifted), it stays in the list as failed with **Retry** and
|
|
**Discard**. Discarding abandons only the code write: the value it set keeps
|
|
rendering.
|
|
|
|
Edits that only affect rendering (narration text, overlay files, render
|
|
options) preview and export immediately. Edits that change the capture itself
|
|
(record options, interaction timings, on-screen text, the language set) are
|
|
badged **applies after next recording**: the preview is marked stale until a
|
|
recording runs. A connected machine auto-records once it applies such an edit;
|
|
otherwise trigger a re-record via CI (see the CI setup guide) or ask a developer
|
|
to run `screenci record`. While a connected machine is actively syncing a
|
|
video's source, that video's editing controls lock briefly until it finishes.
|
|
|
|
## Editing in Editor
|
|
|
|
The editor shows the narration, voices, overlays, and render options the video
|
|
uses. Every item is editable: names declared as a blank array start empty and
|
|
wait for content, and values declared in code show their current code value as
|
|
the starting point.
|
|
|
|
Items whose current value still comes from code are marked with a **set in
|
|
code** badge. Editing such an item queues a write-back into your source (applied
|
|
by the next connected machine), so code and editor never drift apart.
|
|
|
|
Pick a language in the sidebar, then choose **Export** to export a new version
|
|
in that language. Exports are per language: switch the language and export
|
|
again to update another localized version. If edits that need a new recording
|
|
are pending and your machine is connected, Export records first and then
|
|
renders. Exported versions appear in the sidebar's **Exported** group, with a
|
|
status glyph while rendering and a marker on the version served at the public
|
|
URL.
|
|
|
|
Saved editor values are applied automatically to every later upload, so CI
|
|
keeps rendering with them. When this happens the CLI prints a line in the
|
|
upload output, so it is visible in CI logs:
|
|
|
|
```
|
|
Editor configuration applied for "Checkout walkthrough".
|
|
```
|
|
|
|
## Recording from the editor
|
|
|
|
The sidebar's **Recording** group collects every way to produce fresh footage:
|
|
|
|
- **Record on your machine**: with `screenci edit` connected, the record menu
|
|
offers "Record <language> on <machine>". This runs a normal local record of
|
|
the open video and language on your machine and syncs the result back.
|
|
- **Record raw preview footage**: records without rendering, refreshing the
|
|
live preview only. This is also what automatic preview re-records use.
|
|
- **Record via CI**: when the project is connected to GitHub, queues the
|
|
project's recording workflow for this video, no local machine needed.
|
|
|
|
A status line under the menu tracks the run ("Recording en on laptop...",
|
|
"Recording synced."). The regular record run lock applies: if another
|
|
`screenci record` is already running on the machine, the request is reported
|
|
back as failed.
|
|
|
|
## Editor narration from code
|
|
|
|
Pass an **array of cue names** to `video.narration(...)` to declare the cue
|
|
keys in code while the narration text, languages, and voices are configured in
|
|
Editor. Chain `.languages([...])` to seed the language list, since there is no
|
|
text in code to infer it from:
|
|
|
|
```ts
|
|
import { video } from 'screenci'
|
|
|
|
video.narration(['intro', 'checkout', 'outro']).languages(['en'])(
|
|
'Checkout walkthrough',
|
|
async ({ page, narration }) => {
|
|
await narration.intro()
|
|
await page.goto('/checkout')
|
|
await narration.checkout.start()
|
|
// ... visible workflow ...
|
|
await narration.checkout.end()
|
|
await narration.outro()
|
|
}
|
|
)
|
|
```
|
|
|
|
The cues behave exactly like cues whose text is defined in code: callable, with
|
|
explicit `start()` and `end()`, and automatic sequencing between consecutive
|
|
cues. TypeScript knows the declared names, so `narration.typo` is a compile
|
|
error.
|
|
|
|
For each cue, Editor exposes the same voice controls available in code (model
|
|
type, style, accent, and pacing) plus a per-cue volume, alongside the narration
|
|
text and language list.
|
|
|
|
On the **first upload** of a video with blank name-only narration, rendering is
|
|
held until someone fills in the narration on the Editor page. The CLI prints
|
|
the hold together with a direct link to Editor:
|
|
|
|
```
|
|
Rendering for "Checkout walkthrough" is on hold. Configure it in Editor:
|
|
https://app.screenci.com/project/<projectId>/video/<videoId>?editor
|
|
```
|
|
|
|
After the video has been configured once, subsequent uploads reuse the saved
|
|
Editor configuration and render automatically.
|
|
|
|
To supply the text from code instead, pass a plain object. The object takes the
|
|
same shapes as before, either content-major (`{ intro: 'Welcome' }`) or
|
|
language-major (`{ en: { intro: 'Welcome' }, fi: { intro: 'Tervetuloa' } }`):
|
|
|
|
```ts
|
|
import { video } from 'screenci'
|
|
|
|
video.narration({ intro: 'Welcome', checkout: 'Add an item to the cart.' })(
|
|
'Checkout walkthrough',
|
|
async ({ page, narration }) => {
|
|
await narration.intro()
|
|
await page.goto('/checkout')
|
|
await narration.checkout()
|
|
}
|
|
)
|
|
```
|
|
|
|
Because a plain object already carries the narration text, it is **not held**
|
|
on the first upload: it renders straight away from the code values, while
|
|
staying editable so editors can change it later. Once a cue is edited in
|
|
Editor, that Editor value wins and the code value never clobbers it. A blank
|
|
array declaration carries no text, so it is still held until someone fills it
|
|
in. See [Narration](/docs/guides/narration) for the full narration API.
|
|
|
|
## Narration media from Editor
|
|
|
|
Any narration entry in Editor can use an uploaded media file instead of
|
|
synthesized speech, the web equivalent of a code narration cue's
|
|
`{ media: './intro.mp4' }` entry. Switch a cue's entry from **Text** to
|
|
**Media**, upload an `.mp4` file, and optionally provide a subtitle used for
|
|
captions.
|
|
|
|
This works per language, so one language can use an uploaded recording while
|
|
the others keep text-to-speech.
|
|
|
|
### Media subtitles
|
|
|
|
A media narration entry can carry an optional subtitle. When you leave it
|
|
blank, captions are generated automatically from the speech in the uploaded
|
|
file. When you provide one, that text is used instead. Either way, captions are
|
|
timed from the detected speech, so they appear only while the line is actually
|
|
spoken (not during any leading silence or music).
|
|
|
|
## Editor overlays from code
|
|
|
|
Pass an **array of overlay names** to `video.overlays(...)` to declare the
|
|
names in code while the files and display options are configured in Editor. To
|
|
start from code values instead, pass an object (the same overlay shapes as
|
|
always, content-major or language-major): the code values are used until the
|
|
overlay is edited in Editor, after which the Editor value wins. The declared
|
|
names are exposed through the injected `overlays` fixture:
|
|
|
|
```ts
|
|
import { video } from 'screenci'
|
|
|
|
video.overlays(['intro', 'logo'])(
|
|
'Product demo',
|
|
async ({ page, overlays }) => {
|
|
await overlays.intro()
|
|
await page.goto('/dashboard')
|
|
await overlays.logo()
|
|
}
|
|
)
|
|
```
|
|
|
|
To supply the file and placement from code instead, pass an object:
|
|
|
|
```ts
|
|
video.overlays({ logo: { path: 'assets/logo.png', width: 288 } })
|
|
```
|
|
|
|
Calling a controller marks the point in the timeline, exactly like before. The
|
|
file (`.svg`, `.png`, or `.mp4`), full-screen mode, overlay duration for
|
|
images, and audio level for videos are all editable on the Editor page. The
|
|
audio level is a linear-gain slider: `1` (the default) plays the video at its
|
|
natural level, `0` mutes it, and values above `1` boost it (up to `4`). Video
|
|
overlays also have **speed** and **time** controls: speed plays the clip faster
|
|
or slower (a multiplier), and time fits it to a target playback duration in ms.
|
|
Set at most one. TypeScript knows the declared names, so `overlays.typo` is a
|
|
compile error.
|
|
|
|
Like blank narration, the first upload of a video that declares overlays by
|
|
name only is held until every declared overlay has a file configured in Editor.
|
|
The CLI prints a direct link. Later uploads reuse the saved configuration. See
|
|
[Overlays](./overlays.md) for how overlays behave on the timeline.
|
|
|
|
## Editor render and record options
|
|
|
|
Render and record options declared in code are the **starting point**; web
|
|
edits override them:
|
|
|
|
```ts
|
|
import { video } from 'screenci'
|
|
|
|
// Code values: Editor starts from these. An Editor edit wins from then on.
|
|
// Declare per video (supports per-language overrides via { default, <lang> }):
|
|
video.renderOptions({ recording: { size: 0.85 } })
|
|
video.renderOptions({ output: { aspectRatio: '9:16' } })
|
|
|
|
// Or declare nothing: Editor starts from the system defaults.
|
|
```
|
|
|
|
There is no separate deferral: every video's render options are managed on the
|
|
Editor page whether or not code declares any. Options you never touch in
|
|
Editor keep following the code values (and the system defaults beneath them),
|
|
so tuning a value in code still takes effect on later uploads as long as that
|
|
value has not been edited in the web app.
|
|
|
|
Render options are applied when the version renders:
|
|
|
|
```ts
|
|
import { video } from 'screenci'
|
|
|
|
video('Product demo', async ({ page }) => {
|
|
await page.goto('/dashboard')
|
|
})
|
|
```
|
|
|
|
Record options (aspect ratio, quality, fps) work the same way but change the
|
|
captured viewport and encode, so Editor edits to them take effect on the
|
|
**next recording**, not when you click **Export**. They are fetched before the
|
|
recording runs and applied to that capture (later uploads reuse the saved
|
|
values). The Recording options section shows this
|
|
reminder inline with a **Re-record this video** button:
|
|
|
|
```ts
|
|
video.recordOptions({ fps: 30 })('Product demo', async ({ page }) => {
|
|
await page.goto('/dashboard')
|
|
})
|
|
```
|
|
|
|
These options combine with the per-feature declarations and `.each()` like any
|
|
other per-video configuration:
|
|
|
|
```ts
|
|
import { video } from 'screenci'
|
|
|
|
video.recordOptions({ fps: 30 }).narration(['intro']).overlays(['logo'])(
|
|
'Product demo',
|
|
async ({ page, narration, overlays }) => {
|
|
await narration.intro()
|
|
await page.goto('/dashboard')
|
|
await overlays.logo()
|
|
}
|
|
)
|
|
```
|
|
|
|
The recorded **language set** is managed separately via `video.languages(...)`:
|
|
see [Editor languages from code](#editor-languages-from-code) below. There is no
|
|
`recordOptions.languages`.
|
|
|
|
## Editable timeline actions
|
|
|
|
Interaction timings, zoom options, speed blocks, and pauses can be edited from
|
|
the web timeline, without hand-editing code: each saved edit is written into
|
|
the sources for you and picked up by the next record.
|
|
|
|
Every interaction is editable from the web, whether its values come from
|
|
package defaults or from explicit options in code. Its identity is the
|
|
captured locator description (for example `getByRole(button, name=Save)`)
|
|
plus its position on the timeline. Code is the single source of truth: while
|
|
`screenci edit` is connected, each edit you save in the editor is codegen'd
|
|
straight into the `.screenci.ts` sources (keyed by the action's `editId`
|
|
slug), so the code always shows the current values and the next record simply
|
|
runs from code.
|
|
|
|
Cursor-move fields (`move.duration`/`move.speed`, `move.easing`, `move.curve`,
|
|
`move.curviness`, `move.delayAfter`), action durations, and pre-action pauses
|
|
are all written as the matching option on the `editId`-stamped call. The
|
|
cursor path's curve can be edited visually in the preview by dragging its
|
|
bezier handles.
|
|
|
|
Manual `zoomTo(...)` calls and `scrollIntoViewIfNeeded()` also appear on the
|
|
editor's "Zooms & scrolls" row with editable `easing`, `duration`, `amount`,
|
|
and `centering` fields.
|
|
|
|
The main editable action forms:
|
|
|
|
```ts
|
|
import { autoZoom, speed } from 'screenci'
|
|
|
|
// Editable block: the multiplier is owned by the web editor (defaults to 1).
|
|
// The editId identity slug is stamped automatically when an edit session
|
|
// starts; you can also set it yourself.
|
|
await speed(async () => { ... }, { editId: 'intro-speedup' })
|
|
|
|
// Without an editId yet, the block is identified by its timeline position
|
|
// until the next edit session stamps one.
|
|
await speed(async () => { ... })
|
|
|
|
// Explicit: the multiplier comes from code (a web edit rewrites this call).
|
|
await speed(3, async () => { ... })
|
|
|
|
// Bare autoZoom stays fully web-editable, starting from the package defaults.
|
|
await autoZoom(async () => { ... })
|
|
|
|
// Web-editable pause: defaults to 0ms until edited in the web timeline.
|
|
await page.waitForTimeout()
|
|
|
|
// Explicit pause: the duration comes from code (a web edit rewrites it).
|
|
await page.waitForTimeout(500)
|
|
```
|
|
|
|
Pointer actions also expose a web-owned `sleepBefore` field (default 0): the
|
|
SDK sleeps that long after the previous event before the cursor starts
|
|
moving, pushing the action later on the timeline. In the editor, dragging a
|
|
bar's left edge sets it, and the pause shows as a leading "sleep" part of the
|
|
bar.
|
|
|
|
Dragging a whole interaction along the timeline absorbs into the recorded
|
|
`waitForTimeout` next to it: moving it later grows the preceding sleep (and
|
|
moving it earlier shrinks it), rewriting the `waitForTimeout(<ms>)` argument in
|
|
code so the timeline matches what the next record will produce. When an edit
|
|
leaves two `waitForTimeout` calls back-to-back in the source (nothing but sleeps
|
|
between them), they collapse into a single `waitForTimeout` whose duration is
|
|
their sum.
|
|
|
|
Recordings always run purely from code: nothing is fetched or overridden at
|
|
record time. After each upload the timeline is reconciled against what was
|
|
actually recorded, so new actions appear in place and removed actions
|
|
disappear.
|
|
|
|
## Web-authored events
|
|
|
|
Render-affecting events can also be ADDED and MOVED from the web timeline,
|
|
without hand-editing code: hides, speedups, time remaps, narration cues,
|
|
overlays, and recording changes (resize/hide/show). Interactions are
|
|
different on purpose: a click or tap always stays where the test code performed it, and
|
|
only its parameters (durations, sleeps) are editable.
|
|
|
|
Everything the timeline adds is one unified edit record keyed to a call
|
|
position, and it is codegen'd into the sources the moment it is saved (via
|
|
the connected `screenci edit` session). A newly added event appears on the
|
|
timeline as a pending item until the next record confirms it.
|
|
|
|
A web-authored event can be deleted again: select it and press **Delete** or
|
|
**Backspace**, or right-click it and choose **Delete**. Deleting removes the
|
|
edit from both the editor and the source (the same path "Reset all" uses).
|
|
Recorded interactions are code-owned and cannot be deleted this way.
|
|
|
|
Events are added in two ways:
|
|
|
|
- **The Add effect popover** (the "+" on a timeline row) creates a narration
|
|
cue, overlay, camera zoom, speedup, hide, time remap, or recording change,
|
|
anchored to the interaction(s) you pick.
|
|
- **Directly on the Recording row**: toggle **split mode** (the scissors) and
|
|
click the recording to cut it, or drag a section's edge to hide footage from
|
|
either end. Right-clicking a section offers remove (hide), split at the
|
|
current time, reset trim, merge with the neighbor, and quick speed
|
|
(0.5x/2x/4x) and time-remap presets.
|
|
|
|
Every web-placed or web-moved event is positioned by **call position**: which
|
|
editable action it sits after (or, for a span, the run of actions it brackets),
|
|
plus any timing gap as a plain millisecond sleep. The editor snaps to the
|
|
identity of a known action (its stable `editId` slug) rather than to
|
|
wall-clock time, so positions survive re-records whose real durations drift.
|
|
|
|
- A point event (a narration cue, an overlay, a recording resize) is stored
|
|
as "after action X, with an optional `waitForTimeout(ms)` gap before it".
|
|
- A span event (hide, speed, time) is stored as the run of actions it brackets:
|
|
"from action X until action Y", with optional gap sleeps at each edge.
|
|
- A zoom is stored as the run of interactions it wraps, with a lead-in and hold
|
|
expressed as sleeps inside the block.
|
|
|
|
When you drag an event just before an upcoming click, the editor glues it to
|
|
that click by making it the action the event sits before, with a `waitForTimeout`
|
|
gap. There is no free offset field: everything lands in a gap between known
|
|
actions or brackets a known run of actions.
|
|
|
|
Each edit is applied to code the moment it is saved: the dev session locates
|
|
the call site by editId and writes the call-position statement into the
|
|
source. An edit that cannot be applied fails the codegen request and the
|
|
editor reverts the optimistic value instead of dropping it silently. The
|
|
failure carries a typed reason plus a message, so the editor toast says what
|
|
to fix: `unknown-edit-id`, `ambiguous-edit-id`, `inside-control-flow`,
|
|
`unstamped-action`, `loop-repeat`, `unsupported-field`, `invalid-edit`,
|
|
`unresolved-import` (the effect function needs a named import from
|
|
'screenci'), `unknown-video`, `app-managed`, or `unsupported-shape`.
|
|
|
|
Aliased imports are supported throughout: a file that does
|
|
`import { autoZoom as az } from 'screenci'` has its `az(...)` wraps
|
|
recognised, updated, and unwrapped like the canonical name, and codegen reuses
|
|
the alias when inserting new calls.
|
|
|
|
## Effects in code: block wrappers and gap sleeps
|
|
|
|
Everything the web timeline can place, code expresses directly as calls in the
|
|
linear timeline. There are no declarative "placed" helpers and no anchors or
|
|
offsets: an effect's position is simply where its call sits in the test body,
|
|
and timing gaps are plain `await page.waitForTimeout(ms)` sleeps.
|
|
|
|
Render-time spans (hide, speed, time) and camera zooms are **block wrappers**
|
|
that bracket the interactions they cover. Lead-in and hold are sleeps inside
|
|
the block:
|
|
|
|
```ts
|
|
video('Checkout', async ({ page }) => {
|
|
await page.getByRole('button', { name: 'Submit' }).click({ editId: 'submit' })
|
|
|
|
// Hide a loading flicker that appears 250ms after the click, for 500ms.
|
|
await page.waitForTimeout(250)
|
|
await hide(async () => {
|
|
await page.waitForTimeout(500)
|
|
})
|
|
|
|
// Play a stretch of steps at 3x.
|
|
await speed(3, async () => {
|
|
await page.getByRole('button', { name: 'Next' }).click({ editId: 'next' })
|
|
await page
|
|
.getByRole('button', { name: 'Confirm' })
|
|
.click({ editId: 'confirm' })
|
|
})
|
|
|
|
// Fit a block to exactly 400ms of output.
|
|
await time(400, async () => {
|
|
await page
|
|
.getByRole('tab', { name: 'Receipt' })
|
|
.click({ editId: 'receipt' })
|
|
})
|
|
|
|
// Zoom the camera into a click: lead in 400ms BEFORE it and hold 600ms
|
|
// after it. The camera target comes from the mouse positions recorded
|
|
// inside the block.
|
|
await autoZoom(async () => {
|
|
await page.waitForTimeout(400) // lead-in before the first inner action
|
|
await page.getByRole('button', { name: 'Save' }).click({ editId: 'save' })
|
|
await page.waitForTimeout(600) // hold after the last inner action
|
|
})
|
|
})
|
|
```
|
|
|
|
To place an effect a fixed time after an interaction, put a
|
|
`waitForTimeout(ms)` right after that interaction and then the effect. To lead a
|
|
zoom in before a click, open the `autoZoom` block earlier and lead in with a
|
|
sleep as its first inner line. The block's first and last actions define its
|
|
window; you never compute an absolute offset. See
|
|
[Camera and zooming](./camera-and-zooming.md) for `autoZoom`, `zoomTo`, and
|
|
`resetZoom`.
|
|
|
|
Point effects that DO happen at call time (a narration cue, an overlay) are
|
|
just imperative calls in the timeline, paced by ordinary sleeps:
|
|
|
|
```ts
|
|
await page.getByRole('button', { name: 'Stats' }).click({ editId: 'stats' })
|
|
// Start the narration 800ms after the click.
|
|
await page.waitForTimeout(800)
|
|
await narration.stats()
|
|
```
|
|
|
|
Rule of thumb: gaps are `waitForTimeout` sleeps, render-time spans and zooms
|
|
are block wrappers over the interactions they cover, and narration/overlay
|
|
cues are plain calls placed where you want them in call order. The web editor
|
|
shows this same linear timeline, and editor edits are codegen'd into these
|
|
same call-position statements, keyed by each action's `editId`.
|
|
|
|
### Splitting and trimming the recording from the web editor
|
|
|
|
The web timeline has a scissors mode: clicking the recording track cuts it at
|
|
that instant. A bare split is stored as a zero-width `hide` span edit. It is
|
|
editor-only state: codegen never writes an empty `hide(async () => {})` into
|
|
code, so an untouched split just stays editable on the web.
|
|
|
|
A cut snaps to where it will actually land, and the guide line (plus the live
|
|
preview, when paused) tracks that snapped point rather than the raw cursor. A
|
|
click inside an interaction cannot split it mid-action, so it snaps to the
|
|
nearer edge (the gap before or after it), and a second cut in a spot already
|
|
taken is refused. A cut left of the first interaction (over the footage leading
|
|
into it) anchors to that interaction with a backward lead instead of a forward
|
|
gap sleep; once such a span reaches code it opens with a leading
|
|
`waitForTimeout` over that lead-in.
|
|
|
|
Dragging a split's edges inward swallows footage (and the interactions in it)
|
|
into the hide; the span edit is re-anchored to whole interactions, with
|
|
`waitForTimeout` sleeps preserving any partial gap on both sides. Dragging back
|
|
out restores the footage. Once the trimmed span reaches code, it is a regular
|
|
`hide(...)` block.
|
|
|
|
### Removing a code block from the web editor
|
|
|
|
A block carrying an `editId` (`hide(fn, { editId: 'setup' })`, and likewise
|
|
`speed`/`time`) can be removed from the web editor (merge two recording
|
|
sections, reset a trim). This sends a `blockRemoveEdit` targeting the block's
|
|
editId; the codegen channel unwraps the block in source, keeping the wrapped
|
|
calls (any `waitForTimeout` pacing inside survives as plain gap sleeps).
|
|
Blocks without an editId get one stamped automatically when an edit session
|
|
starts, so every block becomes web-removable.
|
|
|
|
### Splitting a camera zoom in two
|
|
|
|
An `autoZoom` bracket on the Zooms row can be split into two back-to-back
|
|
brackets from the web editor: enter split mode (the scissors) and click the
|
|
zoom at the interaction boundary where it should break. A web-added
|
|
(pending) zoom is split by rewriting its own edit record. A code-authored
|
|
`autoZoom` is split through the codegen channel: the editor sends a
|
|
`blockRemoveEdit` for the original bracket's `editId` (which now unwraps
|
|
`autoZoom` blocks, not only `hide`/`speed`/`time`) plus two `zoomEdit`s over
|
|
the two interaction sub-runs, each carrying the original zoom options
|
|
(`amount`/`duration`/`easing`/`centering`) so the halves are identical apart
|
|
from their time. The unwrap is ordered before the two re-wraps in one sync
|
|
pass, so the result is two sibling `autoZoom(...)` blocks. Because this
|
|
rewrites the source, splitting a code zoom needs a connected `screenci edit`
|
|
session; with no machine connected the editor declines rather than storing a
|
|
deferred edit. A zoom framing a single interaction cannot be split.
|
|
|
|
Overlays and narration cues are not yet splittable from the web editor: their
|
|
placements are stored as points (a start position, not a code-level span), so
|
|
there is no duration to divide. Splitting those remains a source edit.
|
|
|
|
### Actions inside `hide()`
|
|
|
|
Instrumented actions inside a `hide()` run raw (no cursor animation) and emit
|
|
no input events, but each one records a small `hiddenAction` marker
|
|
(`{ type: 'hiddenAction', timeMs, action, matcher? }`) in the recording data.
|
|
Renderers ignore these markers; the web editor uses them to know what a hide
|
|
was suppressing.
|
|
|
|
## How edits reach code
|
|
|
|
Code is the single source of truth, and the loop is a single step:
|
|
|
|
1. **Connect.** Run `screenci edit` in the project. The startup handshake
|
|
brings every managed video up to date, then the machine serves the editor.
|
|
2. **Edit in the web timeline.** Each saved edit arrives over the dev channel
|
|
as a codegen request and is written into the `.screenci.ts` sources
|
|
immediately, via static analysis (the TypeScript parser), no agent
|
|
involved. Each edit locates its call site by the exact `editId` slug and
|
|
writes the call-position statement: an option value on the stamped call, a
|
|
`narration.x()` / overlay / presentation call (with a `waitForTimeout`
|
|
gap), or an `autoZoom` / `hide` / `speed` / `time` block bracketing the
|
|
right run of interactions. An edit either applies by editId or its section
|
|
is locked (a loop or branch) and the request fails, reverting the edit in
|
|
the editor.
|
|
3. **Record.** Recordings always run purely from code, so what you see on the
|
|
next record is exactly what the sources say.
|
|
|
|
Because the web timeline and code share one linear model, a codegen'd edit
|
|
inserts the same call you would have written by hand.
|
|
|
|
### Formatting codegen edits
|
|
|
|
After an edit is written, the CLI formats the changed file with your
|
|
project's own Prettier install. `screenci init` enables this by scaffolding a
|
|
minimal `.prettierrc` (2-space indent, single quotes, no semicolons, matching
|
|
the generated examples) and installing `prettier` in the project. Formatting
|
|
runs only when both are present: edit `.prettierrc` to change the style, or
|
|
delete it (or uninstall `prettier`) to keep the raw codegen output. A
|
|
formatting failure never fails the edit; the unformatted change is written
|
|
and a warning is logged.
|
|
|
|
## Action identity: editId
|
|
|
|
Every editable action can carry a stable, human-readable identity slug in
|
|
code, e.g. `.click({ editId: 'click1' })` or
|
|
`autoZoom(fn, { editId: 'autoZoom1' })`. The `screenci edit` startup handshake
|
|
stamps missing slugs automatically after a recording, allocating numbers from
|
|
`.screenci/edit-ids.json` (commit it; numbers are never reused and stamped ids
|
|
are never removed). With an editId, the action's stable key IS the slug: edits
|
|
keep matching across re-records even after refactors, moved lines, or locator
|
|
changes, and codegen locates the call site by the exact slug instead of
|
|
heuristics.
|
|
|
|
The slug is the action's display name on the editor timeline, and it can be
|
|
renamed there: the rename is codegen'd by replacing the slug's string literal
|
|
in code.
|
|
|
|
Because the slug IS the identity, two distinct actions must never share one. A
|
|
copy-pasted `editId` silently merges both into a single identity (the second
|
|
looks like a loop repeat and its edits cannot reach code). Static analysis
|
|
guards against this automatically: before recording, and during the
|
|
`screenci edit` startup handshake and its codegen apply, any slug found at two
|
|
or more distinct call sites is resolved by keeping the first occurrence and
|
|
re-stamping the rest with fresh slugs (allocated from `.screenci/edit-ids.json`,
|
|
so they never collide with an existing id). A genuine loop (one call site that
|
|
runs repeatedly) is a single occurrence in source and is left untouched.
|
|
|
|
editId is optional until edits need to reach code. Actions without one keep
|
|
the matcher-based identity (locator description + occurrence) for display, but
|
|
codegen never guesses at their call sites: their edits cannot apply until the
|
|
dev startup handshake stamps them. An action that executes more than once in a
|
|
recording (a loop) gets keys like `click1#1` for the repeat executions; those
|
|
sit in a locked section that cannot be expressed as code options and are not
|
|
editable.
|
|
|
|
## What is editable from the web
|
|
|
|
Every recorded action carries identity metadata, so the timeline covers:
|
|
|
|
- **Interactions**: all pointer actions (click, fill, tap, check, select,
|
|
hover, selectText, dragTo), with per-part timing (`sleepBefore`, move
|
|
duration, pre-press pause, typing/hover/drag durations).
|
|
- **Camera**: `autoZoom()` blocks, `zoomTo()`, `resetZoom()`,
|
|
`scrollIntoViewIfNeeded()`.
|
|
- **Pacing**: `speed()` blocks (multiplier), `time()` blocks (target
|
|
duration), `page.waitForTimeout()` delays, and named `hide()` spans
|
|
(visible, read-only). `speed`, `time` and `hide` all accept an optional
|
|
name as their first argument for a stable identity.
|
|
- **Presentation**: `resizeRecording`/`hideRecording`/`showRecording` (size,
|
|
duration) and `redact()` mask styling
|
|
(color, radius, css).
|
|
- **Hard borders**: `page.goto` navigations are recorded and shown as
|
|
full-height borders. Their duration is app time: never editable, and
|
|
timing edits cannot cross them.
|
|
|
|
The editor can also ADD events without hand-editing code: hides, speedups,
|
|
time remaps, and recording changes,
|
|
each placed by call position (after a known action, or bracketing a run of
|
|
actions) with any gap expressed as a `waitForTimeout` sleep.
|
|
|
|
## Option panels and narration text reach code too
|
|
|
|
The editor's option panels are codegen'd the same way as timeline edits while
|
|
`screenci edit` is connected (the studio config keeps working as the instant
|
|
preview and the offline fallback):
|
|
|
|
- **Render options** (recording size and roundness, background, aspect ratio,
|
|
quality, mouse size/style/motion blur, keyboard shortcut display, narration
|
|
box styling, shadow, crop) are merged into the video's
|
|
`.renderOptions({...})` builder call. The call is appended to the chain when
|
|
the video has none yet; existing keys are updated in place and unrelated
|
|
keys are left untouched.
|
|
- **Record options** are merged into `.recordOptions({...})` the same way and
|
|
trigger a preview re-record, since they change recorded behavior.
|
|
- **Narration text** is merged into the `video.narration(...)` declaration:
|
|
a new cue key is added, an existing value replaced, and per-cue volume is
|
|
written as the `{ cue, volume }` object form (a plain text edit never
|
|
upgrades a string cue to an object, and editing the text of an object cue
|
|
keeps its other keys). Editing a non-default language converts a flat
|
|
(content-major) declaration to the language-major form: the existing values
|
|
move under `default` verbatim and the edited language gets its own
|
|
sub-object.
|
|
|
|
Every editor edit is codegen'd: it is written into your `.screenci.ts` sources
|
|
through the connected `screenci dev` machine, and fails if no machine is
|
|
connected. There is no web-side edit store. Uploaded media (narration voices
|
|
and recorded audio, cloned-voice samples) is downloaded to local editor files
|
|
on the dev machine and referenced from code.
|
|
|
|
Loop repeats stay locked: an action that runs more than once from a single
|
|
call site (keys like `click1#1`) cannot be edited per execution, in the editor
|
|
or through codegen. Edit the first iteration or the code itself.
|
|
|
|
## Undoing web edits
|
|
|
|
Edits live in your sources, so undoing one is a code change: revert the file
|
|
in git (or edit it by hand) and record again. There is no separate web edit
|
|
layer to reset.
|
|
|
|
## Editor languages from code
|
|
|
|
> **Set `mode`, `locales`, and `browserLocale` correctly in code up front.** The
|
|
> editor can add languages (by writing them into `video.languages([...])`), but
|
|
> it cannot yet edit `mode`, `locales`, or `browserLocale`. Give them their final
|
|
> values now (via `video.languages({ languages, mode, locales, browserLocale })`).
|
|
|
|
The recorded language set is the **union** of the code set declared with
|
|
`video.languages([...])` and any language keys used by the narration
|
|
declaration (overlays are shared across languages). When you add a language to a
|
|
narrated video, the editor offers to auto-translate the existing narrations into
|
|
it, or start it with empty placeholders. The **Languages** section on the Editor
|
|
page shows the
|
|
current set and lets you add a language; adding one writes it into your
|
|
`video.languages([...])` declaration in code (a new `.languages([...])` call is
|
|
added when the video has none) through the connected `screenci dev` machine,
|
|
then records:
|
|
|
|
```ts
|
|
import { video } from 'screenci'
|
|
|
|
// Records en and fi. Adding a language in the editor extends this array.
|
|
video.narration({ en: { intro: 'Hi' } }).languages(['en', 'fi'])(
|
|
'Product tour',
|
|
async ({ page, narration }) => {
|
|
await narration.intro()
|
|
await page.goto('/dashboard')
|
|
}
|
|
)
|
|
```
|
|
|
|
To set the capture options too, pass a config object:
|
|
|
|
```ts
|
|
video.languages({ languages: ['en', 'fi'], mode: 'shared' })
|
|
```
|
|
|
|
The config accepts the same `languages`, `mode`, `locales`, and `browserLocale`
|
|
fields. As noted above, the editor can edit the language set but not
|
|
`mode` / `locales` / `browserLocale` yet, so set those to their final values
|
|
here.
|
|
|
|
Adding a language records a fresh pass for it: the new pass reuses the existing
|
|
capture and the new narration. Because the language set changes the captured
|
|
recording itself (unlike narration text and overlays, applied at render time),
|
|
adding a language always requires a new recording pass. See
|
|
[Languages](./languages.md) for the full language API.
|
|
|
|
## Action parameter tracking and overrides
|
|
|
|
Every instrumented Playwright action (`click`, `fill`, `pressSequentially`,
|
|
`tap`, `check`, `uncheck`, `selectOption`, `hover`, `dragTo`, `selectText`,
|
|
`scrollIntoViewIfNeeded`) records which option values it used, for example
|
|
`move.duration`, `move.speed`, `move.easing`, `move.delayAfter`, `position`,
|
|
`noWaitAfter`, `duration`, and `dragSteps`, and whether each value was set
|
|
explicitly at the call site or came from a default. This provenance is written
|
|
into the uploaded recording data (`actionParams` in `data.json`), so the
|
|
backend and Editor can present the parameters for editing.
|
|
|
|
Editing a parameter in the web editor writes it into the call site as an
|
|
explicit option (via the connected `screenci edit` session), whether the value
|
|
previously came from code or from a default. The recording always runs with
|
|
whatever the code says.
|
|
|
|
The SDK also exports `ACTION_PARAM_DEFAULTS`, the default value of every
|
|
tracked option per action method, so integrations can tell an edit that merely
|
|
restates the default from a real change and offer "reset to default".
|
|
|
|
## Migrating from `editable()`
|
|
|
|
The `editable()` helper has been removed. Everything is editable in the web app
|
|
by default now, so the wrapper is no longer needed:
|
|
|
|
| Before | After |
|
|
| ----------------------------------------------- | -------------------------------------- |
|
|
| `editable(['intro'])` | `['intro']` |
|
|
| `editable({ intro: 'Hi' })` | `{ intro: 'Hi' }` |
|
|
| `video.languages(editable())` | `video.languages()` |
|
|
| `video.languages(editable(['en', 'fi']))` | `video.languages(['en', 'fi'])` |
|
|
| `video.languages(editable({ mode: 'shared' }))` | `video.languages({ mode: 'shared' })` |
|
|
| `use({ recordOptions: editable({ fps: 30 }) })` | `video.recordOptions({ fps: 30 })` |
|
|
| `use({ renderOptions: editable() })` | `video.renderOptions({ default: {} })` |
|
|
|
|
A bare array still declares blank editor-owned names. A plain object now
|
|
supplies code values that are used at record time and remain editable in the
|
|
web app: once edited there, the Editor value wins over the code value.
|