From 490207c00efa9288d007070b2f663b8b5da93180 Mon Sep 17 00:00:00 2001 From: Christian Pillsbury Date: Tue, 14 Jul 2026 12:00:38 -0700 Subject: [PATCH] docs(spf): presentation timeline coordinate model Design doc for the 3-coordinate model (startTime / startDate / startMediaTime) and the timestampOffset relocation approach the reactor + steps implement. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../design/spf/presentation-timeline-model.md | 404 ++++++++++++++++++ 1 file changed, 404 insertions(+) create mode 100644 internal/design/spf/presentation-timeline-model.md diff --git a/internal/design/spf/presentation-timeline-model.md b/internal/design/spf/presentation-timeline-model.md new file mode 100644 index 00000000..29f26b78 --- /dev/null +++ b/internal/design/spf/presentation-timeline-model.md @@ -0,0 +1,404 @@ +--- +status: draft +date: 2026-07-07 +--- + +# Presentation Timeline Coordinate Model + +The coordinate model that lets an SPF engine translate between the three +timelines a streaming presentation lives on — **media**, **presentation**, and +**wall-clock** — and the architecture for **non-zero-PTS `timestampOffset` +relocation** built on it. + +This is a design-in-progress for the `timestampOffset`-relocation approach to +non-zero-PTS (spike branch `spike/spf-non-zero-pts-timestamp-offset`). It is the +"how it's modeled and where it lives" companion to the mechanism decision in +[../../decisions/mse-timestamp-offset.md](../../decisions/mse-timestamp-offset.md) +and the feature framing in +[features/non-zero-pts-support.md](./features/non-zero-pts-support.md). + +--- + +## Problem + +Zero-PTS VOD let one number do everything. The encoded media, the player's +`currentTime`, and the model's `Track.startTime` were all 0-based and identical, +so the loader could compare `currentTime` to `segment.startTime` directly and the +buffer's native PTS matched what the player displayed. + +Non-zero-PTS breaks that identity. A Mux instant clip (`asset_start_time=60`) +encodes its first frame at native PTS ≈ 60s; Apple's bipbop asset starts at 10s. +Now the buffer's native timeline (60→…) and the player's desired 0-based timeline +diverge. Live added a **third** timeline — wall-clock, via +`EXT-X-PROGRAM-DATE-TIME` — and, in doing so, made `Track.startTime` mean +*different things in different engines*: the live anchor rewrites `startTime` onto +native PTS, while VOD keeps it 0-normalized. The same field straddles two +timelines. That ambiguity is the coordinate confusion at the root of the +"non-zero-PTS is hard" problem. + +We want (a) one coherent model of the three timelines, and (b) an architecture +that adds relocation to the engine **without touching the simple case** — no +complexity and no bundle cost for the zero-PTS VOD composition that got us this +far. + +--- + +## Three timelines, one instant + +A timeline is fixed by knowing one instant's coordinate in it. Pick the +**presentation's origin instant** and record its value in each timeline: + +| Field | Timeline | Source | +|---|---|---| +| `startMediaTime` | **media** (encoded/decode) | `tfdt.baseMediaDecodeTime ÷ mdhd.timescale` | +| `startTime` | **presentation** (`currentTime`) | 0 for 0-based product semantics (existing field) | +| `startDate` | **wall-clock** | `EXT-X-PROGRAM-DATE-TIME` (existing field, live) | + +All three advance 1:1 in seconds, so **every translation is pure subtraction** — +no scaling (timescale is already folded into `startMediaTime`): + +```text +timestampOffset = startTime − startMediaTime # presentation − media +wallClock(t) = startDate + (t − startTime) # time-of-day at presentation time t +``` + +`startMediaTime` is the one genuinely new field. The relocation offset is +**derived from the triple, not stored** — storing both would be two +representations that drift (see [conventions/signals.md](./conventions/signals.md)). + +This model **unifies live anchoring and VOD relocation**: the live anchor is the +`(media ↔ wall-clock)` edge; VOD relocation is the `(media ↔ presentation)` edge. +They are two edges of one triangle, which is why the machinery converges (see +[Open questions](#open-questions)). + +### Why `startTime` stays the presentation reference + +`startTime` remaining 0-based (not native media) is *why the relocation approach +is simpler downstream than native-PTS*: with the buffer relocated to 0 via +`timestampOffset`, `currentTime`, `seekable`, and the loader's +`currentTime`-vs-`segment.startTime` window math all stay in one 0-based +coordinate. `startMediaTime` (native) is consumed **only** to compute the offset +at the buffer boundary — the loader never sees it. (The native-PTS branch made +`startTime` native and paid for it with coordinate translation everywhere, +including the initial-load stall.) + +The cost: because all tracks relocate by the shared *min*, a skewed non-primary +track's 0-based `startTime` sits ~skew below its true buffer presentation time +(Apple: ~44ms for video). Sub-frame, only affects coarse load-window planning — +vastly better than native-PTS's full-origin (60s) mismatch. + +--- + +## One offset, applied polymorphically + +Whatever `startMediaTime` a track is given (own in Tier 1, shared `min` in +Tier 2), the relocation applies the same way — `presentation = native + +timestampOffset`, `timestampOffset = startTime − startMediaTime` — only the +*application* differs: + +```text +buffer sample: presentation = sampleNativePTS + timestampOffset # MSE sets sb.timestampOffset +text cue: cueFinal = cueNative + timestampOffset # we compute it + cueNative = LOCAL + MPEGTS/90000 (X-TIMESTAMP-MAP) | absoluteCueTime (no map) +``` + +Text has no `SourceBuffer`, so it *effectively* has a `timestampOffset` applied +as cue arithmetic. This is the live single-anchor rule +([../../decisions/live-presentation-anchor.md](../../decisions/live-presentation-anchor.md)) +in VOD form: **established from the A/V tracks, applied to all tracks including +text.** + +**Shared-`min` across A/V (Tier 2).** When A/V is skewed, the shared +`startMediaTime` is `min` across the audio and video tracks' native origins +(Tier 1 gives each track its own). `min` (not video-primary, not per-track-own): +- keeps every track's earliest **DTS ≥ 0** (relocating by anything larger drives + the lower track negative → Chromium append failure); +- **preserves real A/V skew** (Apple's 44ms audio-lead is retained; relocating + each track to its own 0 would flatten it). + +--- + +## Capability axes (not a tier ladder) + +The "tiers" are really **three orthogonal opt-in axes** — a composition enables +what its platform needs: + +| Axis | Off / simple | On / complex | Driven by | +|---|---|---|---| +| **Relocation** | no `startMediaTime`, offset 0 | read + apply | is the source non-zero-PTS? | +| **Parser** | presumptive (first box) | track-id-matched | container packaging (muxed `clcp`/extra track?) | +| **Coordination** | per-type own (opt-out) | `min`-reduce across A/V (default) | is A/V actually skewed? | + +They cross freely — muxed-captions-but-aligned (track-id + single), or +separate-file-A/V-but-skewed (presumptive + `min`). The parser axis is already +built as two tree-shakeable exports (`media/mp4/timestamp-origin.ts`, +committed `cf8aaca45`): `readFirstMediaTimescale`/`readFirstBaseMediaDecodeTime` +(presumptive) vs `findMediaTrack`/`readBaseMediaDecodeTime` (track-id). + +--- + +## Architecture: discover → derive → apply, established by a reactor + +A per-source reactor, **`establishStartMediaTime`**, owns the coordinate +establishment; the byte-level work rides **steps** in a plain config +`messagePipelines` array (`relocationMessagePipelines`) woven into the segment +loader's pipelines. The loader ships a Tier-0 `fetch → dispatch` pipeline and stays +oblivious to relocation; the steps read composition `state` from their call-time +`deps` (no closures, no context), so the reactor and the pipeline steps are +decoupled — **the reactor never touches pipelines**. Enabling relocation = composing +the reactor + supplying the steps as config (+ optionally a `deriveStartMediaTime` +seam); a Tier-0 composition does neither. + +**The `establishStartMediaTime` reactor** has three states, driven by a `monitor`: +`inactive` (no resolved presentation — clears the transient `mediaContainerData` +slot on entry, so each source starts fresh) → `monitoring` (runs the **derive +effect**) → `established` (the selected A/V tracks carry `startMediaTime`; disables +the derive — establish-once, sticky per source, like the live anchor). Selection +signals are **optional/defensive**, so the one reactor composes across video-only / +audio-only / both. Per-source freshness is structural (the `inactive` transition +clears the slot), not a hand-rolled reset. + +- **Discover** (per-type steps). Head-peek steps read the init (`mdhd` timescale) + and the first media segment (`tfdt` baseMediaDecodeTime, plus that segment's + 0-based `startTime`) and write them into `mediaContainerData`, a + `Signal>` + **keyed by track type** (`'video'`/`'audio'`) — one init+media pair per type + suffices, and ABR rungs of a type share the entry. It's a **two-source** read + across two appends, but the slot *is* the shared state (init writes `timescale`, + the first media segment writes `baseMediaDecodeTime`+`segmentStartTime`), so the + steps are independent — no shared closure, no self-discrimination. Writes are + **synchronous** RMW of disjoint keys (the sync-merge invariant keeps this clear + of the #1746 hazard). Runs on the *fetched* stream, so transport stays pure + `fetchBytes`. `segmentStartTime` is recorded because the origin is + `baseMediaDecodeTime/timescale − segmentStartTime` — the first *loaded* segment + isn't necessarily the 0th (non-zero initial `currentTime`, live/DVR). +- **Derive** (reactor effect). Watches `mediaContainerData` (+ selection) and, via + the injected **`deriveStartMediaTime`** seam, writes the settled per-type + `startMediaTime` onto the `Track`s (each type's value stamped on every track of that + type). The seam is **pure** — + `(mediaContainerData, ctx) => Record` — `undefined` + means "not ready yet"; the effect is the sole writer of the field. It's the + **one coordination knob**, and the default `deriveSharedMinStartMediaTime` collapses + what were "Tier 1" and "Tier 2" into a single reduce: relocate every track by the + `min` across the *selected* A/V origins. This subsumes per-type — aligned A/V → + `min` equals each origin, skewed A/V → `min` keeps every DTS ≥ 0 *and* preserves the + real skew (per-type would flatten it), single-type → its own origin. It returns + `undefined` until every selected type is discovered (the shared-`min` barrier). The + same value goes to the model here and to the buffer via the tier-agnostic apply. No + Presentation-level field is needed — the min is denormalized across the per-type + entries. `derivePerTypeStartMediaTime` remains as the barrier-free opt-out for a + composition that knows its A/V is aligned. +- **Apply** (per-track step). A stamp step relocates via `timestampOffset = + −startMediaTime`, which the SourceBufferActor applies to + `SourceBuffer.timestampOffset`. The apply is **tier-agnostic**: it runs the *same* + `deriveStartMediaTime` seam the reactor uses, over the shared `mediaContainerData` + slot — so the buffer offset always equals the model's stamped `startMediaTime`, and + it's robust to `established` + late tracks (the slot persists and discover gates + per-type; the model value may not be re-stamped once the reactor goes sticky). It's + **async with an `awaitDefined` holdback**: per-type resolves at once (own origin + discovered earlier in the same pipeline); shared-`min` waits until every selected + A/V origin is in (the barrier), filled by the other type's discover step — no + deadlock, since discover precedes the awaiting stamp. A **liveness guard** short- + circuits first: if this type's own origin was never discovered (TS / containerless / + a mock/0-PTS-no-`tfdt` source) it leaves the append native and does not wait. A + derived `0` also leaves it native (setting `timestampOffset` at all can ripple). + Text-cue relocation reuses the model value the reactor stamped (see the text step). + +### Consume: `startMediaTime` lives on the model + +The settled `startMediaTime` lands per-track on the **CMAF-HAM `Track`**, a peer +of `startTime` (presentation) and `startDate` (wall-clock). It's a coordinate base +value that *defines the timeline relationships*, so it belongs on the model, not a +parallel slot; `timestampOffset` stays derived (`startTime − startMediaTime`), +never stored. + +The **churn** — partial per-track reads across appends — stays in the transient +`mediaContainerData` slot and never touches `presentation`. Only the *settled* +value reaches the model, written by the derive effect as **sole writer** of the +field. That write does share `presentation`'s existing multi-writer situation +(#1746, addressed at the presentation-ownership level); we accept that rather than +distort the model to dodge it. This is also where relocation and the live anchor +**converge** — both are "an establishment unit writing coordinate base values onto +tracks" (the anchor writes `startTime`/`startDate`, relocation writes +`startMediaTime`) — the flagged eventual dedup. + +### Branch-free always-present actors + +The pieces present in *every* composition carry **no relocation vocabulary at +all** — not even a no-op'd seam. Relocation lives entirely in the reactor and its +injected steps; the Tier-0 pipeline is literally `[fetch, dispatch]`: + +- **SegmentLoaderActor**: owns only the invariant skeleton (`fetchStep`, + `dispatchStep`, in-flight bookkeeping, between-step abort checks) and a + `messagePipelines` factory that defaults to `fetch → dispatch`. It does not + fetch-whole, parse, wait, or know what a `timestampOffset` is. +- **SourceBufferActor**: sets `sb.timestampOffset` only when the append meta + carries one, idempotent-guarded (`meta.timestampOffset != null && + sb.timestampOffset !== meta.timestampOffset`) so re-stamping the constant offset + on later appends is a no-op. Absent = untouched. Apply-only. + +All tier variation collapses to **the reactor and its injected steps + derive +seam**; the loader and buffer actors are shared, and Tier 0 imports no relocation +code. + +--- + +## Key decisions + +Documented because they were debated. + +### `startTime` stays the presentation reference; add `startMediaTime` (media) + +**Decision:** `startTime` remains 0-based (presentation); add `startMediaTime` +(native media origin) as the new field. `startDate` stays wall-clock. + +**Alternatives:** +- *Make `startTime` honest (native media).* Clean three-field correspondence, but + ripples through the parser (stop 0-normalizing), the loader (translate every + `currentTime`-vs-`startTime` comparison), and live's `startDate` math — i.e. it + *is* the #1746 model cleanup. + +**Rationale:** Relocation keeps everything downstream 0-based, so presentation is +the *correct* timeline for `startTime` here; the native origin is needed only to +derive the offset. Keeps the loader coordinate-consistent and the simple case +untouched. The honest-`startTime`-everywhere convergence (which would also +reconcile live's native `startTime`) is a separate, named future effort. + +### Relocation is a per-source reactor; `startMediaTime` on the model via a pure derive seam + +**Decision:** relocation is the `establishStartMediaTime` reactor (per-source +lifecycle), not a config bundle of loose signals. Discovery and apply are loader +pipeline steps; **derive** is the reactor's effect, driven by a **pure injected +`deriveStartMediaTime` seam** that maps `mediaContainerData` → per-track +`startMediaTime`. The churn lives in the transient `mediaContainerData` slot; the +settled value lands on `Track` (the effect is its sole writer). + +**Alternatives:** *a behavior with a hand-rolled reset effect* (the reactor's +entry/exit gives per-source reset + teardown for free, structurally); *the reduce +as a `computed`* (can't write the model — and putting `startMediaTime` on the +model is the point, per the consume decision); *keeping the transient state +module-local rather than on `state`* (it's coordinate state the model consumes, so +it belongs on `state`, defined through the reactor). + +**Rationale:** the reactor matches the per-source lifecycle relocation actually +has (and that `setupBufferActors` already models), makes stale-write races +structural rather than guarded, and isolates the one tier difference into a pure, +testable seam. + +### Shared-`min` origin (the Tier-2 `deriveStartMediaTime`) + +**Decision:** the coordination axis is entirely the `deriveStartMediaTime` seam. +Tier 1 writes each track its own origin; Tier 2 writes the `min` across the +selected A/V origins onto **every** track. `startMediaTime` stays per-track on +`Track` in both tiers — Tier 2 denormalizes the min across the per-track slots, so +no Presentation-level field is required and **apply is unchanged** across tiers. + +**Alternatives:** *per-track-own for skewed A/V* (flattens real A/V skew, lossy); +*video-primary* (the field norm — VHS/hls.js — but they sidestep negative-DTS via +transmux/offset math we don't have); *a Presentation-level shared value* (not +needed — the seam denormalizing onto tracks keeps the read path uniform). + +**Rationale:** `min` is the only choice that keeps every DTS ≥ 0 *and* preserves +real inter-track skew; expressing it as a swap of the derive seam means Tier 2 is +one function change with no movement in discover, apply, or wiring. + +### Offset derived, not stored; two parsers split for tree-shaking + +`timestampOffset` is a `computed` from the model triple, never a stored field. +The presumptive vs track-id parsers are separate exports (not one +optional-selector function) so a caption-free platform tree-shakes the +track-selection machinery (~37% smaller, verified). Application rides optional +per-op append metadata so the always-present actors stay branch-free. + +### Offset applied via append meta, not a dedicated message (revisitable) + +**Decision:** the offset is carried on each media segment's append meta +(`meta.timestampOffset`) and applied by the SourceBufferActor's +`appendSegmentTask` (idempotent-guarded). A relocating composition's `stampOffset` +pipeline step writes it onto the meta before `dispatch` (absent in Tier 0, whose +pipeline has no such step; the step may be async — see +["One offset, applied polymorphically"](#one-offset-applied-polymorphically)). +Setting the offset is **not** its own SourceBuffer message today. + +**Alternative:** a dedicated `set-timestamp-offset` SourceBuffer message the +loader schedules as its own task — cleaner separation of buffer configuration +from append payload, and the natural shape once each actor's *messages* are +decoupled from the concrete *Tasks* they translate into (segment-loader +`planTasks`/`makeLoadTask`/`scheduleAll`; SourceBufferActor `messageTaskFactories`). + +**Rationale:** the offset is constant per source, so meta-carried + idempotent +apply costs nothing and needs no new protocol. And meta-per-segment *generalizes*: +HLS discontinuities and DASH multi-period re-base `timestampOffset` per period, and +carrying it on each append gets correct in-order application for free via the +`SerialRunner` — a standalone message would have to be interleaved into the append +stream to match. "Set once" is the special case we happen to be in. + +**Left open:** promoting offset-setting to its own message remains a valid future +move; the mid-stream offset-*change* case (discontinuities / multi-period) is the +likely forcing function, though even then meta-per-segment may still win. Nothing +in the current seam precludes the switch — this is a deliberate "not yet," not a +closed door. + +--- + +## Open questions + +- ~~**Generalize the live establishment behavior.**~~ *Resolved (mechanism).* VOD + relocation is a **sibling** of `anchor-presentation-timeline`, not literally it: + a per-source establishment unit (`establishStartMediaTime` reactor) writing a + coordinate base value onto tracks, with the tier logic in a pure derive seam + rather than a `computed`. The two share the "establish-once, apply-to-all" + *shape* (and both write per-track base values → the #1746 multi-writer surface). + Whether they should be **one unit** — the anchor writing `startDate`/`startTime` + and relocation writing `startMediaTime` folded into a single establisher — is + the remaining, larger dedup, still future. See + [../../decisions/live-presentation-anchor.md](../../decisions/live-presentation-anchor.md). +- **Barrier liveness.** For Tier 2, `deriveStartMediaTime` returns `undefined` + until the selected A/V origins are all present, and the apply-side + `awaitDefined` holds each first append until then — so audio erroring, disabled, + or absent must not block forever. The holdback needs a bound (timeout / + audio-disabled short-circuit inside the seam's "when"); both VHS and hls.js + special-case this. +- **Holding the first segment across the wait.** The first media segment is held + (buffered) while the offset resolves, then appended; steady-state streaming is + untouched. The first-chunk peek is validated to contain the `moof` with a + ~60–600× margin (first chunk ≥128KB, `moof` ≈0.2–2KB), so no streaming + box-peeker is needed — but the hold's interaction with preempt/replan needs + care. +- ~~**`startMediaTime` storage granularity.**~~ *Resolved.* Per-track on `Track` + (the self-describing triple), in both tiers — Tier 2 denormalizes the shared + `min` across the per-track slots rather than introducing a presentation-level + field. Promotion to a `Presentation`-level base value is a possible future move + if a real de-dup need appears, but Tier 2 does not force it. +- **Text-only sources.** No A/V `tfdt` to establish from — but the + `X-TIMESTAMP-MAP` `MPEGTS` *is* a media-timeline reference, so text could + self-establish. Deferrable special path. +- ~~**Discovery doesn't belong in the fetch abstraction.**~~ *Resolved.* Discover + is now a head-peek discover step run on the *fetched* byte stream inside the + loader, not a fetch decorator — transport stays pure `fetchBytes`, and parsing + mp4 boxes / writing the raw values into `mediaContainerData` sits where it + belongs (a content concern). It keeps the per-track, byte-first, no-double-fetch + properties. +- **Convergence to honest `startMediaTime` everywhere** (the "A" option above), + which would let live and VOD share one `startTime` semantic. Its own effort. + +--- + +## See also + +- [presentation-modeling.md](./presentation-modeling.md), + [live-presentation-modeling.md](./live-presentation-modeling.md) — the data + model this extends; #1746 (the concurrently-RMW'd `presentation` hazard) is why + the churn stays in the transient `mediaContainerData` slot. +- [../../decisions/mse-timestamp-offset.md](../../decisions/mse-timestamp-offset.md) + — the mechanism decision (native-PTS default; relocation for the 0-based cases). +- [../../decisions/live-presentation-anchor.md](../../decisions/live-presentation-anchor.md), + [../../decisions/live-timeline-anchoring.md](../../decisions/live-timeline-anchoring.md) + — the live anchor this generalizes. +- [features/non-zero-pts-support.md](./features/non-zero-pts-support.md) — the + feature framing. +- `packages/spf/src/media/mp4/` — the committed box parser (presumptive + + track-selected). +- `.claude/plans/spf-non-zero-pts-timestamp-offset-spike.md` — the spike's + running notes; OSS prior-art survey (buffer-whole is universal; + establish-then-gate cross-track) captured in agent memory.