diff --git a/internal/decisions/end-of-stream-av-skew-recovery.md b/internal/decisions/end-of-stream-av-skew-recovery.md new file mode 100644 index 00000000..f5444a7d --- /dev/null +++ b/internal/decisions/end-of-stream-av-skew-recovery.md @@ -0,0 +1,102 @@ +--- +status: decided +date: 2026-07-13 +--- + +# End-of-Stream Recovery for Skewed / Tiny-Final-Segment A/V + +## Decision + +Make a complete VOD reach native `ended` (and therefore loop) reliably, even when the +audio and video tracks don't end at exactly the same time, via **two narrow changes** in +the two behaviors that already own the end of playback: + +1. **`end-of-stream` reactor** — add slack to the "playhead reached the last segment" + gate: fire `endOfStream()` once `currentTime >= lastSegStart − LAST_SEGMENT_REACHED_SLACK` + (0.5 s) rather than `>= lastSegStart` exactly. +2. **`recover-end-stall` behavior** (new, DOM) — on the `waiting` event, if the + MediaSource is `'ended'`, the stream is finite, and the playhead is within + `endStallNudgeWindow` (default 0.2 s) of the reachable buffered end, set + `currentTime = duration` to force native `ended`. + +Both are event-driven; neither polls. `recover-end-stall` reads only `mediaElement` + +`mediaSource`. The window/slack are config-tunable. + +## Context + +Chrome hangs at the end of the Apple `bipbop_adv_example_hevc` VOD (a source with a ~44 ms +A/V PTS skew): the playhead freezes a few frames short of the end and native `ended` never +fires, so playback stalls and loop never re-triggers. Investigation (measured in the +`spf-non-zero-pts` sandbox) found **two** compounding causes: + +- **A tiny final segment deadlocks the EOS reactor.** Apple's last video segment is ~44 ms + and starts right at the buffered end (`lastSegStart ≈ 600.0`, buffered end ≈ `600.044`). + Chrome paces `currentTime` off the audio clock and freezes the playhead ~50–70 ms short of + the buffered end — i.e. *below* `lastSegStart`. The reactor's `currentTime >= lastSegStart` + gate never opens → `endOfStream()` is never called → the MediaSource stays `'open'` → the + browser keeps the playhead frozen waiting for data/EOS that never comes. A seek past + `lastSegStart` (e.g. to `duration`) breaks the deadlock. +- **Even once `endOfStream()` fires, the playhead still freezes short of `duration`.** With + MS `'ended'`, Chrome still stops ~50–70 ms short (the audio clock can't advance past the + shorter track's end), so `currentTime` never reaches `duration` and `ended` never fires. + Nudging `currentTime = duration` fires it immediately. + +Empirical findings that shaped the design: + +- The freeze gap (playhead-stop → reachable buffered end) is stably ~50–70 ms (measured 52, + 58, 62, 71 ms), with ~20 ms non-deterministic jitter in the exact stop position. The + reachable end (`buffered.end(last)` = `min(video, audio)`, the audio/pacing track) is + stable; `duration` (= `max`, the video end) varies run-to-run with ABR rendition. +- `waiting` fires at the freeze with ~0 ms latency (measured −3.2 ms vs a rAF sampler), so a + poll would only add latency — no reason to poll for this. +- A MediaSource reaches `'ended'` **only** via `endOfStream()` (MSE spec); the browser never + does it spontaneously. The manual nudge appeared to "end without EOS" only because the + seek re-triggered our own reactor, which then called `endOfStream()`. +- hls.js handles the same class of stall in its `GapController`: it does **not** trim buffers; + it detects the near-end stall (MS `'ended'` + within 1 s of the edge) and, in a player-layer + event, declares ended. We adapt this to SPF's native-`ended` shape by nudging to `duration`. + +## Alternatives Considered + +- **Trim the longer track's tail to align A/V ends, and set `duration` to the min.** + Prototyped and rejected: trimming *video* by PTS can orphan B-frames (removing frames a + displayable frame depends on), and it doesn't even fix the stall (the freeze persists), and + the mismatch is frame-granular so exact alignment is impossible. It also needs a per-cycle + trim target + re-entry bookkeeping. Adds risk for no benefit once the nudge is in place. +- **Poll for the stall (hls.js's `GapController` 100 ms tick).** Unnecessary here: `waiting` + fires at the freeze with ~0 latency; a poll adds its interval + a stall threshold + (hls.js waits up to `detectStallWithCurrentTimeMs = 1250 ms`). +- **hls.js's "within ~1 s of `duration`" window.** Looser than needed. We gate on proximity to + the reachable buffered end (`buffered.end(last)`), which is tied to real buffered content and + ~5× tighter, sized just above the measured freeze gap. +- **A single fix in one behavior.** Neither alone suffices: without the reactor slack the MS + never reaches `'ended'`; without the nudge the playhead never reaches `duration`. + +## Rationale + +- **Root + residual, in their owners.** The reactor slack fixes the *root* (EOS never firing); + `recover-end-stall` handles the *residual* audio-clock freeze. Each change lives in the + behavior whose concern it is (MediaSource finalization vs. playhead recovery), mirroring + hls.js's separation. +- **Ordering removes the race.** With slack, `endOfStream()` fires as `currentTime` crosses + `lastSegStart − slack` — *before* the freeze — so by the time `waiting` fires the MS is + already `'ended'` and `recover-end-stall`'s gate is satisfied. +- **Grounded constants.** `LAST_SEGMENT_REACHED_SLACK` (0.5 s) and `endStallNudgeWindow` + (0.2 s) both comfortably exceed the measured ~50–71 ms freeze gap. Too-tight risks *missing* + the stall (a permanent hang — worse than the bug), so both bias generous and are tunable. +- **Inert when not needed.** `recover-end-stall` no-ops for live (MS never `'ended'` while + growing) and for streams that end cleanly (no `waiting`); the reactor slack only shifts EOS + slightly earlier near the true end (harmless — the last segment is already appended). + +## Scope + +General EOS robustness for any skewed / short-final-segment A/V — not specific to the +non-zero-PTS relocation work it was discovered alongside. Composed in `engine.ts` and +`engine-audio-only.ts`. + +## Follow-ups + +- The nudge is a single jump to `duration`; if a stream ever freezes *further* short than + `endStallNudgeWindow`, add a bounded retry rather than widening the default blindly. +- `recover-end-stall` currently keys off `waiting`; if a pathological stream froze without a + `waiting`, a one-shot re-check on the MS→`'ended'` transition is the fallback. diff --git a/packages/spf/src/playback/behaviors/dom/end-of-stream.ts b/packages/spf/src/playback/behaviors/dom/end-of-stream.ts index 89e35fe5..a1a51f39 100644 --- a/packages/spf/src/playback/behaviors/dom/end-of-stream.ts +++ b/packages/spf/src/playback/behaviors/dom/end-of-stream.ts @@ -61,11 +61,12 @@ * * # currentTime gate * - * `currentTime` must have reached at least one active track's last - * segment startTime. Prevents `'eos-ready'` entry when a back-buffer - * `remove()` / `appendBuffer()` briefly re-opens the MediaSource while - * the user is mid-stream. HLS rendition time-alignment means any active - * track works as the reference. + * `currentTime` must have reached (within {@link LAST_SEGMENT_REACHED_SLACK}) + * at least one active track's last segment startTime. Prevents `'eos-ready'` + * entry when a back-buffer `remove()` / `appendBuffer()` briefly re-opens the + * MediaSource while the user is mid-stream. HLS rendition time-alignment means + * any active track works as the reference. The slack absorbs the near-end + * playhead freeze (see the constant) so a tiny final segment doesn't deadlock. * * # MS readyState — local subscription * @@ -119,6 +120,18 @@ export interface EndOfStreamContext { type EndOfStreamFsmState = 'preconditions-unmet' | 'eos-ready'; +/** + * Slack (seconds) on the "playhead has reached the last segment" gate. A tiny final + * segment (e.g. Apple's ~44ms last segment) starts right at the buffered end, and the + * browser freezes the playhead ~50–70ms short of that end (its render horizon), so a + * strict `currentTime >= lastSegStart` would never open — deadlocking `endOfStream` + * (the MediaSource stays `'open'`, so the browser keeps the playhead frozen waiting for + * data/EOS that never comes). This slack lets a playhead stalled just short of the final + * segment still finalize. Firing slightly early is harmless: the last segment is already + * appended (the gate above), so no more data is expected. + */ +const LAST_SEGMENT_REACHED_SLACK = 0.5; + function deriveState( presentation: MaybeResolvedPresentation | undefined, mediaSource: MediaSource | undefined, @@ -164,7 +177,8 @@ function deriveState( } } - if (lastSegStart !== undefined && (currentTime ?? 0) < lastSegStart) { + // Slack absorbs the near-end playhead freeze so a tiny final segment doesn't deadlock. + if (lastSegStart !== undefined && (currentTime ?? 0) < lastSegStart - LAST_SEGMENT_REACHED_SLACK) { return 'preconditions-unmet'; } diff --git a/packages/spf/src/playback/behaviors/dom/recover-end-stall.ts b/packages/spf/src/playback/behaviors/dom/recover-end-stall.ts new file mode 100644 index 00000000..fae9d4b3 --- /dev/null +++ b/packages/spf/src/playback/behaviors/dom/recover-end-stall.ts @@ -0,0 +1,121 @@ +/** + * Recover the end-of-stream stall that Chrome exhibits on skewed A/V. After + * `endOfStream`, when the audio and video tracks end a few ms apart (e.g. a source + * with an A/V PTS skew), Chrome's audio-clock-paced playback freezes the playhead + * ~50–70ms short of the reachable buffered end and never fires `ended` — so playback + * hangs at the very end and loop never re-triggers. This behavior watches for the + * `waiting` event that fires at that freeze and, when the MediaSource is `ended` and + * the playhead sits at the reachable buffered end, nudges `currentTime` to `duration` + * to force the native `ended`. + * + * **Event-driven, no poll.** `waiting` fires at the instant the playhead stalls + * (measured ~0ms latency), so there's nothing to gain from polling — and polling would + * add its interval + a stall threshold before reacting. + * + * **Proximity to the *intersection* buffered end** is the discriminator. + * `mediaElement.buffered.end(last)` is already `min(videoEnd, audioEnd)` — the furthest + * point playback can reach — and once `endOfStream` is signalled that's the true content + * end. Requiring the playhead within `endStallNudgeWindow` of it distinguishes the real + * end-of-stream freeze from a mid-stream buffer-hole stall (which sits far from the + * buffered end), so we never skip content. The window must exceed the freeze gap; too + * small would miss the stall (a permanent hang), so the default is generous relative to + * the measured gap and is config-tunable for empirical tuning. + * + * Inert where it shouldn't act: live (the MediaSource never reaches `ended` while the + * window grows; `duration` is `Infinity`) and streams that end cleanly (no `waiting`). + * + * See `internal/decisions/end-of-stream-av-skew-recovery.md`. + */ +import { listen } from '@videojs/utils/dom'; +import { defineBehavior } from '../../../core/composition/create-composition'; +import { effect } from '../../../core/signals/effect'; +import type { ReadonlySignal } from '../../../core/signals/primitives'; + +export interface RecoverEndStallContext { + mediaElement?: HTMLMediaElement | undefined; + mediaSource?: MediaSource | undefined; +} + +export interface RecoverEndStallConfig { + /** + * How close (seconds) the playhead must be to the reachable buffered end for a + * `waiting` to count as the end-of-stream freeze. Must exceed the audio-clock freeze + * gap (~50–70ms measured on Chrome); too small risks missing the stall (a permanent + * hang), so keep a margin. Default {@link DEFAULT_END_STALL_NUDGE_WINDOW}. + */ + endStallNudgeWindow?: number; +} + +/** ~2.8× the measured max freeze gap (71ms) — tight, but with headroom against a miss. */ +export const DEFAULT_END_STALL_NUDGE_WINDOW = 0.2; + +/** + * Whether a `waiting` should be forced to `ended`: the MediaSource is `ended`, the + * stream is finite (not live), playback is active (not paused/seeking/already-ended), + * and the playhead sits within `nudgeWindow` of the reachable buffered end (so it's the + * true end, not a mid-stream buffer hole). Pure — the behavior supplies the live values. + */ +export function shouldForceEnded( + input: { + msEnded: boolean; + durationFinite: boolean; + paused: boolean; + seeking: boolean; + ended: boolean; + currentTime: number; + bufferedEnd: number | undefined; + }, + nudgeWindow: number +): boolean { + const { msEnded, durationFinite, paused, seeking, ended, currentTime, bufferedEnd } = input; + if (!msEnded || !durationFinite || paused || seeking || ended || bufferedEnd === undefined) { + return false; + } + const gap = bufferedEnd - currentTime; + return gap >= 0 && gap < nudgeWindow; +} + +function recoverEndStallSetup({ + context, + config, +}: { + context: { + mediaElement: ReadonlySignal; + mediaSource: ReadonlySignal; + }; + config?: RecoverEndStallConfig; +}): () => void { + const nudgeWindow = config?.endStallNudgeWindow ?? DEFAULT_END_STALL_NUDGE_WINDOW; + + return effect(() => { + const mediaElement = context.mediaElement.get(); + if (!mediaElement) return; + + const onWaiting = () => { + const { buffered } = mediaElement; + const forceEnded = shouldForceEnded( + { + msEnded: context.mediaSource.get()?.readyState === 'ended', + durationFinite: Number.isFinite(mediaElement.duration), + paused: mediaElement.paused, + seeking: mediaElement.seeking, + ended: mediaElement.ended, + currentTime: mediaElement.currentTime, + bufferedEnd: buffered.length > 0 ? buffered.end(buffered.length - 1) : undefined, + }, + nudgeWindow + ); + // Nudge to `duration` → native `ended` (the seeking/ended guards above prevent a + // re-fire while the nudge-seek is in flight, so no latch is needed). + if (forceEnded) mediaElement.currentTime = mediaElement.duration; + }; + + return listen(mediaElement, 'waiting', onWaiting); + }); +} + +export const recoverEndStall = defineBehavior({ + stateKeys: [] as const, + contextKeys: ['mediaElement', 'mediaSource'] as const, + setup: recoverEndStallSetup, +}); diff --git a/packages/spf/src/playback/behaviors/dom/tests/recover-end-stall.test.ts b/packages/spf/src/playback/behaviors/dom/tests/recover-end-stall.test.ts new file mode 100644 index 00000000..474b335e --- /dev/null +++ b/packages/spf/src/playback/behaviors/dom/tests/recover-end-stall.test.ts @@ -0,0 +1,50 @@ +import { describe, expect, it } from 'vitest'; +import { shouldForceEnded } from '../recover-end-stall'; + +// At the end-of-stream freeze: MediaSource ended, finite duration, actively playing, +// and the playhead a few frames short of the reachable (intersection) buffered end. +const atEndStall = { + msEnded: true, + durationFinite: true, + paused: false, + seeking: false, + ended: false, + currentTime: 599.95, + bufferedEnd: 600.0, +} as const; + +const WINDOW = 0.2; + +describe('shouldForceEnded', () => { + it('fires at the end-of-stream freeze (playhead within the window of the buffered end)', () => { + expect(shouldForceEnded(atEndStall, WINDOW)).toBe(true); // gap 0.05 < 0.2 + }); + + it('does not fire mid-content (playhead far from the buffered end)', () => { + expect(shouldForceEnded({ ...atEndStall, currentTime: 300, bufferedEnd: 600 }, WINDOW)).toBe(false); + }); + + it('does not fire until endOfStream is signalled', () => { + expect(shouldForceEnded({ ...atEndStall, msEnded: false }, WINDOW)).toBe(false); + }); + + it('does not fire for live (non-finite duration)', () => { + expect(shouldForceEnded({ ...atEndStall, durationFinite: false }, WINDOW)).toBe(false); + }); + + it('does not fire while paused, seeking, or already ended', () => { + expect(shouldForceEnded({ ...atEndStall, paused: true }, WINDOW)).toBe(false); + expect(shouldForceEnded({ ...atEndStall, seeking: true }, WINDOW)).toBe(false); + expect(shouldForceEnded({ ...atEndStall, ended: true }, WINDOW)).toBe(false); + }); + + it('does not fire with no buffered ranges', () => { + expect(shouldForceEnded({ ...atEndStall, bufferedEnd: undefined }, WINDOW)).toBe(false); + }); + + it('respects the configured window', () => { + const gap015 = { ...atEndStall, currentTime: 599.85, bufferedEnd: 600.0 }; // gap 0.15 + expect(shouldForceEnded(gap015, 0.2)).toBe(true); + expect(shouldForceEnded(gap015, 0.1)).toBe(false); + }); +}); diff --git a/packages/spf/src/playback/engines/hls/engine-audio-only.ts b/packages/spf/src/playback/engines/hls/engine-audio-only.ts index e9119f58..658c2e20 100644 --- a/packages/spf/src/playback/engines/hls/engine-audio-only.ts +++ b/packages/spf/src/playback/engines/hls/engine-audio-only.ts @@ -21,6 +21,7 @@ import { import { deriveCdnPriority } from '../../behaviors/derive-cdn-priority'; import { endOfStream } from '../../behaviors/dom/end-of-stream'; import { loadAudioSegments } from '../../behaviors/dom/load-segments'; +import { recoverEndStall } from '../../behaviors/dom/recover-end-stall'; import { relocationPipelinesFor } from '../../behaviors/dom/relocation-steps'; import { setupAudioBufferActors } from '../../behaviors/dom/setup-buffer-actors'; import { setupMediaSource } from '../../behaviors/dom/setup-mediasource'; @@ -250,6 +251,9 @@ export function createHlsAudioOnlyEngine( // `mediaSource.sourceBuffers` aggregately — composes unchanged with // only audio in scope. endOfStream, + // Force native `ended` if Chrome freezes the playhead short of the buffered end + // after `endOfStream`. Inert for a clean-ending single-track source. + recoverEndStall, // Adapter signal callback. shareSignals, diff --git a/packages/spf/src/playback/engines/hls/engine.ts b/packages/spf/src/playback/engines/hls/engine.ts index 80367c94..4c36802f 100644 --- a/packages/spf/src/playback/engines/hls/engine.ts +++ b/packages/spf/src/playback/engines/hls/engine.ts @@ -43,6 +43,7 @@ import { import { deriveCdnPriority } from '../../behaviors/derive-cdn-priority'; import { endOfStream } from '../../behaviors/dom/end-of-stream'; import { loadAudioSegments, loadTextTrackSegments, loadVideoSegments } from '../../behaviors/dom/load-segments'; +import { recoverEndStall } from '../../behaviors/dom/recover-end-stall'; import { relocatingTextPipelines, relocationPipelinesFor } from '../../behaviors/dom/relocation-steps'; import { seekToLiveEdge } from '../../behaviors/dom/seek-to-live-edge'; import { setupAudioBufferActors, setupVideoBufferActors } from '../../behaviors/dom/setup-buffer-actors'; @@ -297,6 +298,13 @@ export interface SimpleHlsEngineConfig extends ShareSignalsConfig