v10/internal/design/spf/features/mse-mms-pipeline.md

status, date, definition

status	date	definition
implemented	2026-05-20	sketched

MSE / MMS pipeline

The lifecycle that makes a MediaSource a valid driver of the HTMLMediaElement: create + attach the MediaSource, set up per-type SourceBuffers wrapped in actors, propagate presentation duration, and coordinate endOfStream() once playback reaches the appended tail. Spans both standard MediaSource and Safari's ManagedMediaSource.

This doc captures the capability surface: what works, what doesn't, which behaviors / actors / helpers implement it, and how it relates to other features. Segment loading orchestration and buffer flushing are sibling features — this one ends at the lifecycle boundary, not at "data appears in the buffer."

Status

• Composition: createSimpleHlsEngine (HLS VoD)
• Definition depth: sketched — capability surface and implementation footprint documented; MediaSourceActor factoring is an open follow-up tracked under sibling candidates. Audio SourceBuffer flush orchestration is part of multi-language-audio's Tier 2 mid-stream-switching phase, not a separately-scoped feature

Phases of complexity

What's implemented today, organized as platform / capability slices. Each row is a slice that could in principle stand alone; in practice they share the same four behaviors and one actor.

Phase	What	Notes
MediaSource attach / detach lifecycle	Create MediaSource, attach to element, await 'open', publish on context.mediaSource; detach + clear on source reset	setupMediaSource rides resolvePresentation's resolved/unresolved transitions for source resets — direct URL replacement is structural, not a special case
Per-type SourceBuffer + actor setup	One SourceBuffer per type (CMAF A+V), each wrapped in SourceBufferActor + SegmentLoaderActor; gates only on that type's selection + codecs	setupVideoBufferActors / setupAudioBufferActors share setupBufferActors. The behaviors are decoupled — no cross-type coupling in stateKeys
Firefox mozHasAudio cross-type invariant	Both addSourceBuffer calls land in the same runPending iteration so the video buffer exists by the time any append begins	Preserves Firefox's permanent-mozHasAudio=false guard. Carried by SPF effect coalescing + composition order (video registered before audio in the engine). Sandbox repro at apps/sandbox/src/firefox-mse-repro/
ManagedMediaSource (Safari)	preferManaged: true → MMS via srcObject + disableRemotePlayback; standard MSE via createObjectURL otherwise	Same lifecycle shape, different attach surface. preferManaged is hardcoded true today (no config knob)
Initial mediaSource.duration write	Write presentation.duration to mediaSource.duration exactly once per MediaSource, gated on MS open + sourceBuffers idle + clamp ≥ getMaxBufferedEnd	updateMediaSourceDuration; idempotent — leaves any non-NaN value alone. Infinity supported for live
End-of-stream coordination	Call mediaSource.endOfStream() once every active actor's currently-loading track has its last segment appended + the playhead reaches that segment; re-arm on each open → ended → open cycle	endOfStream; sets final duration from getMaxBufferedEnd first to keep the value deterministic against CMAF timestamp drift

What's not implemented

• Mid-stream same-codec buffer flush orchestration — SourceBufferActor accepts remove messages and flushBuffer exists in media/dom/mse/, but no SPF behavior drives flushing on language switch or other mid-stream cleanup. Belongs to multi-language-audio's Tier 2 mid-stream-switching phase, which orchestrates flush on top of this feature's remove-message + flushBuffer primitives.
• changeType() codec-change switching — cross-tick mid-stream track switch after appends begin is "out of scope for this behavior" per setup-buffer-actors.ts. Routes to [5.1-surround-selection] and [hevc-variant-selection] (where the codec change motivates the buffer-recreation or changeType path).
• MediaSourceActor abstraction — endOfStream and updateMediaSourceDuration both subscribe to readyState via onMediaSourceReadyStateChange + a behavior-local signal; both wait for buffers idle. A MediaSourceActor whose snapshot exposes readyState + accepts duration-write / end-of-stream / add-source-buffer messages would coalesce these call sites. Hinted at in both behaviors' JSDoc.
• preferManaged opt-out — hardcoded true. No engine config to force standard MSE on Safari (for testing parity or debugging).
• Continuous live-duration sync — Infinity is supported on the initial write, but updateMediaSourceDuration's "exactly once" contract doesn't re-sync if presentation.duration drifts mid-source. Likely fine for live (one continuous Infinity), but live-stream-support may surface counterexamples.
• setLiveSeekableRange / clearLiveSeekableRange — neither is called today. Live streams under duration === Infinity have an empty HTMLMediaElement.seekable without the explicit setter. Belongs to live-stream-support's Live edge tracking + Terminated state transition phases; lives at this feature's MSE boundary.

Implementation surface

Composition: packages/spf/src/playback/engines/hls/engine.ts — MSE behaviors composed between presentation duration calculation and segment loading. Video buffer setup registered before audio for the Firefox invariant.

Behaviors:

Behavior	File	Responsibility
setupMediaSource	packages/spf/src/playback/behaviors/dom/setup-mediasource.ts	Create + attach MediaSource, await 'open', publish; detach + clear on source reset
setupVideoBufferActors	packages/spf/src/playback/behaviors/dom/setup-buffer-actors.ts	Per-type video buffer + actor setup; sole writer of bandwidthState via createTrackedFetch
setupAudioBufferActors	packages/spf/src/playback/behaviors/dom/setup-buffer-actors.ts	Per-type audio buffer + actor setup; uses plain fetchStream (no bandwidth sampling today)
updateMediaSourceDuration	packages/spf/src/playback/behaviors/dom/update-mediasource-duration.ts	Write mediaSource.duration = presentation.duration once per MS, gated on MS open + buffers idle + spec clamp
endOfStream	packages/spf/src/playback/behaviors/dom/end-of-stream.ts	Drive each open → ended transition once last segments + playhead align; re-arm on cycles

Actor:

Actor	File	Role
SourceBufferActor	packages/spf/src/playback/actors/dom/source-buffer.ts	Serializes append-init / append-segment / remove / batch / cancel via SerialRunner on a single SourceBuffer; snapshot exposes 'idle' / 'updating' for downstream gating (canonical idle gate for endOfStream)

DOM-bound helpers: packages/spf/src/media/dom/mse/

Module	Role
mediasource-setup.ts	createMediaSource({ preferManaged }), attachMediaSource (MMS srcObject + disableRemotePlayback / MSE createObjectURL branch), createSourceBuffer, buildMimeCodec, isCodecSupported, onMediaSourceReadyStateChange, waitForMediaSourceOpen, supportsMediaSource, supportsManagedMediaSource
append-segment.ts	appendSegment — ArrayBuffer + streaming append primitive (used by SourceBufferActor's append tasks)
buffer-flusher.ts	flushBuffer — range removal primitive (used by SourceBufferActor's remove task; multi-language-audio's Tier 2 mid-stream-switching phase consumes via the actor message, not directly)
duration.ts	shouldUpdateDuration, waitForSourceBuffersReady, getMaxBufferedEnd (spec-clamp helper)
end-of-stream.ts	isLastSegmentAppended predicate
mediasource.d.ts	ManagedMediaSource global type augmentation (Safari-only API, not in standard DOM lib)

State slots — reads only. Every MSE behavior has read-only state signatures. Engine state flows in; DOM mutations flow out.

• Reads: presentation (all four behaviors), currentTime (endOfStream), selectedVideoTrackId (setupVideoBufferActors), selectedAudioTrackId (setupAudioBufferActors)

Context slots:

• context.mediaSource — sole writer setupMediaSource. Readers: setupVideoBufferActors, setupAudioBufferActors, updateMediaSourceDuration, endOfStream, loadVideoSegments, loadAudioSegments.
• context.videoBufferActor + context.videoSegmentLoaderActor — sole writer setupVideoBufferActors. Readers: loadVideoSegments, endOfStream.
• context.audioBufferActor + context.audioSegmentLoaderActor — sole writer setupAudioBufferActors. Readers: loadAudioSegments, endOfStream.

DOM-property multi-writer. mediaSource.duration is written by both updateMediaSourceDuration (initial, once-per-MS while NaN) and endOfStream (final, from getMaxBufferedEnd before EOS). Decision domains are non-overlapping — updateMediaSourceDuration's idempotency on non-NaN keeps it out of the EOS path. Not a state-signal multi-writer; the convention from conventions/signals.md doesn't apply directly.

Config surface

This feature has essentially no engine-level config surface today — behaviors read presentation for codecs + duration and operate on context-published resources, with no tuning knobs of their own. preferManaged: true is hardcoded in setupMediaSource. Related engine config (forwardBuffer, backBuffer, bandwidth, quality) flows through setupVideoBufferActors's SegmentLoaderActor construction but belongs to buffer-management (forwardBuffer / backBuffer) and video-abr (bandwidth / quality) respectively.

Verification

• Unit tests:
  • packages/spf/src/playback/behaviors/dom/tests/setup-mediasource.test.ts
  • packages/spf/src/playback/behaviors/dom/tests/setup-buffer-actors.test.ts
  • packages/spf/src/playback/behaviors/dom/tests/update-mediasource-duration.test.ts
  • packages/spf/src/playback/behaviors/dom/tests/end-of-stream.test.ts
  • packages/spf/src/playback/actors/dom/tests/source-buffer.test.ts
  • packages/spf/src/media/dom/mse/tests/* — helper-level coverage (MS/MMS detection, attach branch, duration helpers, EOS predicate)
• Sandbox:
  • apps/sandbox/src/spf-segment-loading/ — main SPF MSE pipeline demo; exercises full lifecycle end-to-end
  • apps/sandbox/src/firefox-mse-repro/ — Firefox mozHasAudio invariant repro; load-bearing for verifying composition order + runPending semantics survive future refactors
  • apps/sandbox/src/simple-hls-html/ / simple-hls-react/ — engine integration through the adapter layer

Open questions

• MediaSourceActor? Both endOfStream and updateMediaSourceDuration carry behavior-local msIsOpen mirrors and buffers-idle waits. An actor that owns the MediaSource (snapshot for readyState; messages for duration-write / end-of-stream / add-source-buffer) would coalesce three call sites and prepare for a future loop-mode (auto-fetch earlier segments mid-ended) where MS and SourceBuffer coordination grows. Hinted at in both behaviors' JSDoc.
• preferManaged as config. Should there be an engine-level opt-out for testing standard MSE on Safari, or remain hardcoded?
• Two-fire endOfStream() on mid-end ABR switches. Accepted today as the price of dropping selectedTrackId dependence in endOfStream. Worth flagging if it ever surfaces downstream issues (e.g., spurious ended events on the element between the two fires).

Related features

• preload-modes — gates this feature indirectly. setupMediaSource rides resolvePresentation's resolved/unresolved transitions, which only flip to resolved once the preload gate (preload !== 'none' or loadActivated) is open.
• capability-probing (candidate) — owns the upstream codec filtering that would prevent createSourceBuffer's late-failure throw from firing in practice. Today's isCodecSupported helper is the seed primitive; capability-probing wraps it into a uniform surface and adds multivariant-level filtering before selection.
• source-replacement — the resolved/unresolved lifecycle setupMediaSource rides is the canonical mechanism for in-place source replacement. Detach-on-state-exit is what makes URL changes work without recreating the engine.
• subtitles — text tracks share the per-type segment-loading FSM but do not touch MSE (no SourceBuffer for text); cleanly separated by media/dom/mse/ not appearing in the text path.
• video-abr — setupVideoBufferActors is the sample producer (createTrackedFetch writes bandwidthState); ABR consumes. Sampling lives here, selection lives there.
• multi-language-audio — its Tier 2 "audio SourceBuffer flush on switch" orchestrates flush on top of this feature's remove-message
  • flushBuffer primitives. The orchestration belongs in multi-language-audio (not a separately-scoped buffer-flushing feature).
• buffer-management — sibling feature for the per-type load-FSM and segment planning that runs on top of the buffers + actors this feature stands up. Sends append-init / append-segment / remove / cancel messages to the SourceBufferActor documented here.
• 5.1-surround-selection (not yet documented, candidate) — cross-codec switching via changeType(); out of scope for this feature's same-codec lifecycle.
• hevc-variant-selection (not yet documented, candidate) — same pattern as 5.1 but for video codec swap.
• live-stream-support — Infinity duration + the EOS picture differ; the "exactly once" duration contract is the spot to revisit. Also the home for the setLiveSeekableRange / clearLiveSeekableRange DOM-exposure surface (see What's not implemented above).
• drm-support (not yet documented, candidate, issue #1411) — key-system readiness would gate MSE setup + append per clusters.md.

Use cases that compose this feature

• audio-only-mode-override (partial — Phase 1 landed) — Phase 1 baseline constituent. Used as-is — MediaSource + endOfStream gate compose unchanged across variants per the uniform-across-tracks discipline (endOfStream reads mediaSource.sourceBuffers aggregately). Verified end-to-end in engine-audio-only.test.ts against both audio-only and mixed-AV manifests.
• video-only-mode-override (coarse) — Phase 1 baseline constituent. Used as-is; the Firefox mozHasAudio cross-type invariant documented here is more pointedly relevant — the variant must produce mozHasAudio=false cleanly under subtractive-audio composition. Phase 1 includes empirical verification.

See also

• presentation-modeling.md — architectural deep-dive on the format-neutral data shape and per-track resolution layer; setup behaviors here gate on isResolvedPresentation from that layer
• text-track-architecture.md — peer architectural deep-dive (different domain, same SPF shape)
• packages/spf/docs/hls-engine.md — full engine composition walkthrough (Stage 5: MSE setup; Stage 8: end-of-stream)
• conventions/behaviors.md — per-type specialization details
• conventions/signals.md — multi-writer slot conventions (relevant for mediaSource.duration as a non-signal multi-writer footnote)