wip(spf): relocate text cues via a composed step pipeline (#6)

Reinstate text-cue relocation (dropped with relocation.ts) and, in the
same pass, rearchitect the text-segment loader onto the v/a loader's
composed-step model so relocation is a pipeline step, not a resolver
wrapper.

Text loader gains TextFrame / TextLoadStep / TextStepDeps /
TextMessagePipelines mirroring segment-loader, with base resolveCuesStep
+ dispatchCuesStep (default: resolveCues -> dispatchCues). makeLoadTask
now just runs the pipeline. The resolver reverts to a pure (url) => cues
host primitive (the text analog of fetchBytes) -- composition-awareness
moves entirely into the injected step, so composition state threads via
compositionDeps into TextStepDeps.

Relocation supplies relocatingTextPipelines (resolveWithMetadata ->
relocateCues -> dispatchCues): cues + X-TIMESTAMP-MAP fetched in
parallel, cues shifted by (mapCorrection - startMediaTime) using the
primary A/V track's startMediaTime (video ?? audio, awaited). The engine
bakes textMessagePipelines, mirroring video/audioMessagePipelines.

Base steps are generic arrow consts (TextStepDeps is contravariant in C
via textTracksActor.send). Sandbox-verified 0-based against the Mux
asset_start_time=60 clip: active cue brackets the 0-based playhead.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Christian Pillsbury
2026-07-10 10:29:35 -07:00
co-authored by Claude Opus 4.8
parent a304067c53
commit 8a9e2911fd
6 changed files with 262 additions and 32 deletions
@@ -45,6 +45,14 @@ your detailed pass; **DEFERRED** = later/open.
`engine-audio-only`) — audio 0-based sandbox-verified
- per-type keying (#3) + non-0th-segment origin `bmdt/ts segmentStartTime` (#4) +
`derivePerTypeStartMediaTime` unit test — unit + end-to-end (start-at-300) verified
- text-cue relocation (#6) — `relocatingTextPipelines` (`resolveWithMetadata →
relocateCues → dispatchCues`) shifts cues by `mapCorrection startMediaTime`, reading
the primary A/V origin (video ?? audio, awaited). Text loader rearchitected onto the
v/a composed-step model (`TextFrame`/`TextLoadStep`/`TextStepDeps`/`TextMessagePipelines`,
base `resolveCuesStep`/`dispatchCuesStep`); resolver reverted to a pure `(url) → cues`
host primitive; composition `state` threaded via `compositionDeps` into `TextStepDeps`;
engine bakes `textMessagePipelines` (mirrors `video`/`audioMessagePipelines`).
Sandbox-verified 0-based (active cue brackets 0-based playhead)
### Architecture (as landed)
- **Steps** (`behaviors/dom/relocation-steps.ts`, `relocationPipelinesFor(type)`): a
@@ -78,9 +86,6 @@ your detailed pass; **DEFERRED** = later/open.
the *raw* discovered origin from `mediaContainerData`, not the reduced
`Track.startMediaTime`. Tier 2's shared-`min` is where stamp reads the reduced
model value — revisit the stamp/`established` interaction then.
- **Text-cue relocation** — DROPPED with `relocation.ts`; captions on a non-zero-PTS
source aren't rebased. Reinstate a relocating `resolveTextTrackSegment` reading the
primary video track's `startMediaTime`.
- **Composition opt-in / tree-shaking** — baked into the standard engine (per decision)
with comment markers; revisit a tree-shakeable opt-in + measure Tier-0 bundle.
@@ -1,4 +1,5 @@
import { createMachineActor, type HandlerContext, type MessageActor } from '../../core/actors/create-machine-actor';
import type { AnySlotMap } from '../../core/composition/create-composition';
import { peek } from '../../core/signals/primitives';
import { SerialRunner, Task } from '../../core/tasks/task';
import {
@@ -57,10 +58,65 @@ export type TextTrackSegmentLoaderActor = MessageActor<
* "Resolve" because the fn covers both network fetch and parse into the
* domain model. Host-agnostic — the concrete resolver (e.g. the
* browser's native VTT resolver) is supplied at engine-assembly time,
* so this actor stays DOM-free.
* so this actor stays DOM-free. A pure `url → cues` primitive (the text
* analog of the v/a loader's `fetchBytes`); composition-awareness lives in
* the injected {@link TextLoadStep}s, not here.
*/
export type TextTrackSegmentResolver<C extends Cue = Cue> = (url: string) => Promise<C[]>;
/**
* A text load in mid-pipeline — the text analog of the v/a loader's `Frame`.
* `resolveCuesStep` fills `cues`; `dispatchCuesStep` sends them. `metadata` is
* opaque header metadata a resolve step may attach for a later step to read
* (e.g. relocation stashes the `X-TIMESTAMP-MAP` correlation here for its rebase
* step). Typed `unknown` so the generic loader stays host-agnostic — the step
* that reads it knows its concrete shape (mirrors `StepDeps.state`).
*/
export interface TextFrame<C extends Cue = Cue> {
readonly op: TextLoadTask;
cues?: C[];
metadata?: unknown;
}
/**
* One stage of a text message pipeline — the text analog of the v/a loader's
* `LoadStep`. Mutates the {@link TextFrame} in place and may be async; the runner
* checks `signal.aborted` before each step and passes the actor's
* {@link TextStepDeps} on every call, so a stateless step (`resolveCuesStep`) is a
* plain value and a step that needs composition signals (relocation's cue rebase)
* reads them from `deps` at call time.
*/
export type TextLoadStep<C extends Cue = Cue> = (
frame: TextFrame<C>,
signal: AbortSignal,
deps: TextStepDeps<C>
) => void | Promise<void>;
/** Per-actor + composition dependencies, passed to each {@link TextLoadStep} on every call — the text analog of `StepDeps`. */
export interface TextStepDeps<C extends Cue = Cue> {
textTracksActor: TextTracksActor<C>;
resolveSegment: TextTrackSegmentResolver<C>;
/**
* The composition deps — full `state`/`context` signal maps + engine `config` —
* passed opaquely so a step that needs composition signals reads them at call
* time (e.g. relocation reading the primary A/V track's `startMediaTime`). Typed
* loose: the loader is a conduit and never reads them; base steps ignore them.
*/
state: AnySlotMap;
context: AnySlotMap;
config: object;
}
/**
* Builds the ordered step list, called **once per actor** (mirrors the v/a loader's
* `MessagePipelines`, but text has a single op type so it's a flat array, not a
* `Record`). The default ({@link DEFAULT_TEXT_MESSAGE_PIPELINES}) is
* `resolveCues → dispatchCues`; a non-zero-PTS composition returns a list that
* inserts a cue-rebase step (see `relocatingTextPipelines`), so the loader stays
* oblivious to relocation.
*/
export type TextMessagePipelines<C extends Cue = Cue> = () => TextLoadStep<C>[];
/**
* Configuration for `createTextTrackSegmentLoaderActor`. Spread over
* `DEFAULT_FORWARD_BUFFER_CONFIG` to override individual forward-window
@@ -68,8 +124,10 @@ export type TextTrackSegmentResolver<C extends Cue = Cue> = (url: string) => Pro
* concern — cues evict by their playhead-relative window at runtime —
* so no `backBuffer` config field.
*/
export interface TextTrackSegmentLoaderActorConfig {
export interface TextTrackSegmentLoaderActorConfig<C extends Cue = Cue> {
forwardBuffer?: Partial<ForwardBufferConfig>;
/** Ordered step pipeline. Defaults to {@link DEFAULT_TEXT_MESSAGE_PIPELINES} (`resolveCues → dispatchCues`). */
messagePipelines?: TextMessagePipelines<C>;
}
// =============================================================================
@@ -82,6 +140,48 @@ interface TextLoadTask {
trackId: string;
}
// =============================================================================
// Steps
// =============================================================================
// Base steps are generic over the cue type `C` (generic arrow consts, not
// `TextLoadStep<Cue>` values): `TextStepDeps` is contravariant in `C` via
// `textTracksActor.send`, so a `Cue`-typed step wouldn't slot into a `VTTCue`
// pipeline. A generic function assigns to any `TextLoadStep<C>` instantiation.
/** Resolve the op's cues (via the injected host primitive) into the frame. The text analog of `fetchStep`. */
export const resolveCuesStep = async <C extends Cue>(
frame: TextFrame<C>,
signal: AbortSignal,
deps: TextStepDeps<C>
): Promise<void> => {
const cues = await deps.resolveSegment(frame.op.segment.url);
if (signal.aborted) return;
frame.cues = cues;
};
/** Dispatch the frame's cues to the TextTracksActor as `add-cues`. The text analog of `dispatchStep`. */
export const dispatchCuesStep = <C extends Cue>(
frame: TextFrame<C>,
_signal: AbortSignal,
deps: TextStepDeps<C>
): void => {
const { op } = frame;
deps.textTracksActor.send({
type: 'add-cues',
meta: {
trackId: op.trackId,
id: op.segment.id,
startTime: op.segment.startTime,
duration: op.segment.duration,
},
cues: frame.cues ?? [],
});
};
/** Tier 0 default: resolve then dispatch. No relocation vocabulary. */
const DEFAULT_TEXT_MESSAGE_PIPELINES = <C extends Cue>(): TextLoadStep<C>[] => [resolveCuesStep, dispatchCuesStep];
/**
* Loads text-track segments for a track and delegates cue management
* to a TextTracksActor. Mirrors the v/a `SegmentLoaderActor` shape (FSM
@@ -112,12 +212,19 @@ interface TextLoadTask {
export function createTextTrackSegmentLoaderActor<C extends Cue>(
textTracksActor: TextTracksActor<C>,
resolveSegment: TextTrackSegmentResolver<C>,
config: TextTrackSegmentLoaderActorConfig = {}
config: TextTrackSegmentLoaderActorConfig<C> = {},
// Composition deps threaded opaquely into each step's `TextStepDeps` (relocation
// reads composition state). The loader never reads them. Defaults empty for
// standalone / base-pipeline use.
compositionDeps: Pick<TextStepDeps<C>, 'state' | 'context' | 'config'> = { state: {}, context: {}, config: {} }
): TextTrackSegmentLoaderActor {
type UserState = Exclude<TextTrackSegmentLoaderActorState, 'destroyed'>;
type Ctx = HandlerContext<UserState, TextTrackSegmentLoaderActorContext, () => SerialRunner>;
const forwardBufferConfig: ForwardBufferConfig = { ...DEFAULT_FORWARD_BUFFER_CONFIG, ...config.forwardBuffer };
const deps: TextStepDeps<C> = { textTracksActor, resolveSegment, ...compositionDeps };
// Built once per actor; default is `resolveCues → dispatchCues`.
const pipeline = (config.messagePipelines ?? DEFAULT_TEXT_MESSAGE_PIPELINES)();
/**
* Translate a load message into an ordered TextLoadTask list based on
@@ -148,27 +255,26 @@ export function createTextTrackSegmentLoaderActor<C extends Cue>(
};
/**
* Wraps a TextLoadTask into a Task that fetches + dispatches `add-cues`.
* Updates `inFlightSegmentId` around the fetch so the load handler can
* make accurate continue/preempt decisions.
* Wraps a TextLoadTask into a Task that runs the op's step pipeline
* (resolve/relocate/dispatch, per the composition's `messagePipelines`).
* Updates `inFlightSegmentId` around the async region so the load handler can
* make accurate continue/preempt decisions, and checks the abort signal before
* each step.
*
* Text degrades gracefully: a step throwing (e.g. a failed segment fetch) is
* logged and swallowed so the runner continues to the next segment — unlike the
* v/a loader, where a failed init must abort the remaining tasks.
*/
const makeLoadTask = (op: TextLoadTask, { getContext, setContext }: Ctx): Task<void> => {
return new Task(async (signal) => {
if (signal.aborted) return;
const frame: TextFrame<C> = { op };
setContext({ ...getContext(), inFlightTrackId: op.trackId, inFlightSegmentId: op.segment.id });
try {
const cues = await resolveSegment(op.segment.url);
if (signal.aborted) return;
textTracksActor.send({
type: 'add-cues',
meta: {
trackId: op.trackId,
id: op.segment.id,
startTime: op.segment.startTime,
duration: op.segment.duration,
},
cues,
});
for (const step of pipeline) {
if (signal.aborted) return;
await step(frame, signal, deps);
}
} catch (error) {
// Graceful degradation: log and continue to the next segment.
console.error('Failed to load text-track segment:', error);
@@ -12,11 +12,19 @@
* carry the `SourceBuffer`-backed actor); the relocation logic itself is byte/signal
* work. Tier 1: `stamp` computes the offset straight from the discovered origin, so
* it's independent of the reactor's derive and works for late tracks.
*
* The text half (`relocatingTextPipelines`) is the same idea for the text-segment
* loader: a `resolveWithMetadata → relocateCues → dispatchCues` pipeline that shifts
* VTT cues onto the same 0-based timeline, reading the primary A/V track's
* `startMediaTime` (the reactor's consumed value) via `deps`.
*/
import type { StateSignals } from '../../../core/composition/create-composition';
import { effect } from '../../../core/signals/effect';
import { peek, type Signal, update } from '../../../core/signals/primitives';
import { resolveVttSegmentMetadata, type TextSegmentMetadata } from '../../../media/dom/text/resolve-vtt-segment';
import { readFirstBaseMediaDecodeTime, readFirstMediaTimescale } from '../../../media/mp4/timestamp-origin';
import type { MediaContainerData } from '../../../media/types';
import { findTrackById } from '../../../media/utils/tracks';
import {
dispatchStep,
fetchStep,
@@ -24,6 +32,7 @@ import {
type MessagePipelines,
type StepDeps,
} from '../../actors/dom/segment-loader';
import { dispatchCuesStep, type TextLoadStep, type TextMessagePipelines } from '../../actors/text-track-segment-loader';
import { peekHead } from '../../primitives/head-peek';
import type { EstablishStartMediaTimeState } from '../establish-start-media-time';
@@ -109,3 +118,80 @@ export function relocationPipelinesFor(trackType: 'video' | 'audio'): MessagePip
'append-segment': [fetchStep, readSegmentOrigin, stampStartMediaTime, dispatchStep],
});
}
// ============================================================================
// TEXT (cue relocation)
// ============================================================================
/** Resolve once `read()` returns a number. No bound: for fMP4 the A/V origin always establishes (0-PTS → 0). */
function awaitDefined(read: () => number | undefined): Promise<number> {
return new Promise((resolve) => {
let stop: (() => void) | undefined;
stop = effect(() => {
const value = read();
if (value !== undefined) {
stop?.();
resolve(value);
}
});
});
}
/**
* Resolve step for the relocation text pipeline. Reuses the injected host resolver
* (`deps.resolveSegment`) for cues and fetches the `X-TIMESTAMP-MAP` header in
* parallel, stashing it on `frame.metadata` for `relocateCuesStep`. Replaces the
* base `resolveCuesStep` (which fetches cues only) — text's native `<track>` parser
* discards the header, so the map needs its own raw-bytes fetch.
*/
const resolveWithMetadataStep: TextLoadStep<VTTCue> = async (frame, signal, deps) => {
const [cues, metadata] = await Promise.all([
deps.resolveSegment(frame.op.segment.url),
resolveVttSegmentMetadata(frame.op.segment.url),
]);
if (signal.aborted) return;
frame.cues = cues;
frame.metadata = metadata;
};
/**
* Relocate step — shifts each VTT cue onto the 0-based presentation timeline:
* `cueFinal = cueNative startMediaTime`, where `startMediaTime` is the primary
* A/V track's origin (selected **video**, else **audio** — the single-anchor rule,
* and defensive like the reactor's optional selection) and `cueNative` folds in the
* `X-TIMESTAMP-MAP` correction (`mpegts/90000 local`) for map-bearing VTT (Apple)
* or is the absolute cue time (no map, e.g. Mux). Text can resolve before A/V
* establishes, so the origin is awaited; fMP4 always establishes it (0-PTS → 0),
* and a text-only source (no A/V selected) simply gets offset 0.
*/
const relocateCuesStep: TextLoadStep<VTTCue> = async (frame, signal, deps) => {
if (!frame.cues?.length) return;
const state = deps.state as unknown as StateSignals<EstablishStartMediaTimeState>;
const startMediaTime = await awaitDefined(() => {
const primaryId = state.selectedVideoTrackId.get() ?? state.selectedAudioTrackId.get();
if (primaryId === undefined) return 0;
const presentation = state.presentation.get();
return presentation ? findTrackById(presentation, primaryId)?.startMediaTime : undefined;
});
if (signal.aborted) return;
const { timestampMap } = (frame.metadata as TextSegmentMetadata | undefined) ?? {};
const mapCorrection = timestampMap ? timestampMap.mpegts / 90000 - timestampMap.local : 0;
const delta = mapCorrection - startMediaTime;
if (delta !== 0) {
for (const cue of frame.cues) {
cue.startTime += delta;
cue.endTime += delta;
}
}
};
/**
* Relocation text pipeline — the text analog of `relocationPipelinesFor(type)`.
* `resolveWithMetadata` (cues + `X-TIMESTAMP-MAP`) → `relocateCues` (shift by the
* primary A/V origin) → `dispatchCues`.
*/
export const relocatingTextPipelines: TextMessagePipelines<VTTCue> = () => [
resolveWithMetadataStep,
relocateCuesStep,
dispatchCuesStep,
];
@@ -18,12 +18,13 @@
* `config` so this behavior owns the DOM-bound part of the text-track
* pipeline.
*/
import { defineBehavior } from '../../../core/composition/create-composition';
import type { AnySlotMap, Behavior } from '../../../core/composition/create-composition';
import { effect } from '../../../core/signals/effect';
import type { ReadonlySignal, Signal } from '../../../core/signals/primitives';
import { createTextTracksActor } from '../../actors/dom/text-tracks';
import {
createTextTrackSegmentLoaderActor,
type TextMessagePipelines,
type TextTrackSegmentLoaderActor,
type TextTrackSegmentLoaderActorConfig,
type TextTrackSegmentResolver,
@@ -36,14 +37,24 @@ export interface TextTrackActorsContext {
textTrackSegmentLoaderActor?: TextTrackSegmentLoaderActor | undefined;
}
export interface TextTrackActorsConfig extends TextTrackSegmentLoaderActorConfig {
export interface TextTrackActorsConfig extends Pick<TextTrackSegmentLoaderActorConfig<VTTCue>, 'forwardBuffer'> {
resolveTextTrackSegment: TextTrackSegmentResolver<VTTCue>;
/**
* Ordered text step pipeline, mapped to the loader's `messagePipelines`. Named
* with the `text` domain prefix to mirror the v/a `video`/`audioMessagePipelines`
* composition-config slots. Defaults (in the loader) to `resolveCues → dispatchCues`.
*/
textMessagePipelines?: TextMessagePipelines<VTTCue>;
}
function setupTextTrackActorsSetup({
state,
context,
config,
}: {
// Forwarded opaquely into the loader's steps (relocation reads composition state);
// this behavior owns no state of its own (stateKeys: []).
state: AnySlotMap;
context: {
mediaElement: ReadonlySignal<TextTrackActorsContext['mediaElement']>;
textTracksActor: Signal<TextTrackActorsContext['textTracksActor']>;
@@ -59,7 +70,9 @@ function setupTextTrackActorsSetup({
const textTrackSegmentLoaderActor = createTextTrackSegmentLoaderActor(
textTracksActor,
config.resolveTextTrackSegment,
{ forwardBuffer: config.forwardBuffer }
{ forwardBuffer: config.forwardBuffer, messagePipelines: config.textMessagePipelines },
// Composition deps forwarded into each step (relocation reads the primary A/V origin).
{ state, context, config }
);
context.textTracksActor.set(textTracksActor);
context.textTrackSegmentLoaderActor.set(textTrackSegmentLoaderActor);
@@ -73,8 +86,20 @@ function setupTextTrackActorsSetup({
});
}
export const setupTextTrackActors = defineBehavior({
// Manual `Behavior` literal (like `end-of-stream`): no state of its own
// (`stateKeys: []`), but the setup forwards the composition `state` to the resolver
// opaquely. A literal (not `defineBehavior`) so `stateKeys: []` can coexist with a
// setup that reads `state`.
export const setupTextTrackActors: Behavior<
Record<never, never>,
{
mediaElement: ReadonlySignal<TextTrackActorsContext['mediaElement']>;
textTracksActor: Signal<TextTrackActorsContext['textTracksActor']>;
textTrackSegmentLoaderActor: Signal<TextTrackActorsContext['textTrackSegmentLoaderActor']>;
},
TextTrackActorsConfig
> = {
stateKeys: [],
contextKeys: ['mediaElement', 'textTracksActor', 'textTrackSegmentLoaderActor'],
setup: setupTextTrackActorsSetup,
});
};
@@ -112,7 +112,11 @@ function setupLoadTextTrackCues(initialState: TextTrackSegmentLoadingState, init
// targeting dormant / activation behavior override this explicitly.
const state = makeState({ preload: 'auto', ...initialState });
const context = makeContext(initialContext);
const setupCleanup = setupTextTrackActors.setup({ context, config: { resolveTextTrackSegment: resolveVttSegment } });
const setupCleanup = setupTextTrackActors.setup({
state,
context,
config: { resolveTextTrackSegment: resolveVttSegment },
}) as () => void;
const reactor = loadTextTrackSegments.setup({ state, context });
const cleanup = () => {
reactor.destroy();
@@ -43,7 +43,7 @@ import {
import { deriveCdnPriority } from '../../behaviors/derive-cdn-priority';
import { endOfStream } from '../../behaviors/dom/end-of-stream';
import { loadAudioSegments, loadTextTrackSegments, loadVideoSegments } from '../../behaviors/dom/load-segments';
import { relocationPipelinesFor } from '../../behaviors/dom/relocation-steps';
import { relocatingTextPipelines, relocationPipelinesFor } from '../../behaviors/dom/relocation-steps';
import { seekToLiveEdge } from '../../behaviors/dom/seek-to-live-edge';
import { setupAudioBufferActors, setupVideoBufferActors } from '../../behaviors/dom/setup-buffer-actors';
import { setupMediaSource } from '../../behaviors/dom/setup-mediasource';
@@ -54,8 +54,9 @@ import { trackCurrentTime } from '../../behaviors/dom/track-current-time';
import { trackLoadTriggers } from '../../behaviors/dom/track-load-triggers';
import { updateMediaSourceDuration } from '../../behaviors/dom/update-mediasource-duration';
// Non-zero-PTS relocation (spike): remove this import, the composed reactor, the
// `video/audioMessagePipelines` finalConfig entries, the `mediaContainerData` state
// slot, and the `deriveStartMediaTime` config field to drop relocation entirely.
// `video/audio/textMessagePipelines` finalConfig entries, the `mediaContainerData`
// state slot, and the `deriveStartMediaTime` config field to drop relocation entirely
// (text then falls back to the plain `resolveVttSegment` resolver).
import { type DeriveStartMediaTime, establishStartMediaTime } from '../../behaviors/establish-start-media-time';
import { type ParsePresentation, resolvePresentation } from '../../behaviors/resolve-presentation';
import { resolveAudioTrack, resolveTextTrack, resolveVideoTrack } from '../../behaviors/resolve-track';
@@ -355,6 +356,9 @@ export function createSimpleHlsEngine(
// composes always.
reschedule: config.reschedule ?? delayedReschedule(mediaPlaylistReloadDelay),
resolveTextTrackSegment: config.resolveTextTrackSegment ?? resolveVttSegment,
// Non-zero-PTS relocation (spike): the text pipeline rebases cues onto the
// relocated 0-based timeline. Remove `textMessagePipelines` to drop text relocation.
textMessagePipelines: relocatingTextPipelines,
resolveDuration: config.resolveDuration ?? getResolvedSelectedTrackDuration,
parsePresentation: config.parsePresentation ?? parseMultivariantPlaylist,
addSubtitlesTracksToMedia: config.addSubtitlesTracksToMedia ?? addSubtitlesTracksToMedia,
@@ -423,8 +427,8 @@ export function createSimpleHlsEngine(
// segment-loader pipelines to context. MUST precede `setup*BufferActors`
// so the pipelines are published before the loaders read them. Remove this
// one line (+ the import, the `mediaContainerData`/`*MessagePipelines`
// slots, and the `deriveStartMediaTime` config) to drop relocation and
// test the Tier-0 baseline / bundle size.
// slots including `textMessagePipelines`, and the `deriveStartMediaTime`
// config) to drop relocation and test the Tier-0 baseline / bundle size.
establishStartMediaTime,
// ─────────────────────────────────────────────────────────────────────