v10/internal/design/spf/features/audio-playback.md

status, date, definition

status	date	definition
implemented	2026-05-20	sketched

Audio playback

Today's single-rendition audio playback in the HLS engine. The engine parses audio renditions from the multivariant playlist, picks one via the default picker at source load, fetches its media playlist, sets up an audio SourceBuffer, and loads segments. The selected rendition is locked for the lifetime of the source — mid-stream switching belongs to multi-language-audio (coarse, not yet implemented).

Symmetric with video-abr (video) and subtitles (text) in the per-track-type capability set. Most of the implementation surface is owned architecturally by other features (mse-mms-pipeline, buffer-management, source-replacement); this doc captures what's audio-specific.

Status

• Composition: createSimpleHlsEngine (HLS VoD)
• Definition depth: sketched — capability surface documented; language-aware selection exists in media/primitives/ but isn't wired (see What's not implemented)

Phases of complexity

Capability slices around today's audio playback contract.

Phase	What	Notes
Audio rendition recognition	parseMultivariantPlaylist surfaces audio renditions from #EXT-X-MEDIA:TYPE=AUDIO lines with language, name, default, autoselect, channels, codecs, uri	Engine consumes at most one today; parsing handles multiple — the foundation multi-language-audio builds on
Default audio rendition selection	selectAudioTrack runs the default picker once on 'presentation-resolved' entry. Default picker today is pickFirstTrackId (first track in the audio selection set) — language-unaware	Config-overridable via SelectAudioTrackConfig.picker. The language-aware pickAudioTrack exists in media/primitives/ but isn't wired — see What's not implemented
Audio media playlist resolution	resolveAudioTrack (sibling of resolveVideoTrack / resolveTextTrack, shared setupTrackResolution helper) fetches the selected rendition's media playlist on entry; aborts on source un-resolve via state-bound AbortController	Same shape as video / text resolution
Audio SourceBuffer + actor setup	setupAudioBufferActors creates the audio SourceBuffer + SourceBufferActor + SegmentLoaderActor. Uses plain fetchStream (no bandwidth sampling — there's no audio ABR yet)	Owned architecturally by mse-mms-pipeline; the Firefox mozHasAudio cross-type invariant lives in that feature's documentation
Audio segment loading	loadAudioSegments dispatches the per-type 4-state segment-load FSM ('preconditions-unmet' / 'dormant' / 'metadata-only' / 'full-range') consuming (preload, loadActivated). Same gate behavior as video	Owned architecturally by buffer-management
Source-change clearance	On source un-resolve, selectedAudioTrackId clears, audio actors tear down, in-flight playlist fetch aborts. New source's audio rendition picked fresh on the next resolve	Owned architecturally by source-replacement

What's not implemented

• Multi-rendition recognition + programmatic selection + mid-stream switching — covered by multi-language-audio (partial, sketched — Tier 1 + most of Tier 2 implemented). Default selection now uses pickAudioTrack's three-tier picker (preferredAudioLanguage → DEFAULT=YES → first-track), so preferredAudioLanguage config takes effect. Tier 2 programmatic selection via userAudioTrackSelection filter and same-codec mid-stream switching with next-segment-boundary flush also implemented. Persistence and A/V sync policy refinements deferred.
• Audio ABR — covered by audio-abr. setupAudioBufferActors uses plain fetchStream; the bandwidth-sampling createTrackedFetch isn't wired into the audio fetch path. The sampling primitive exists (used by video); audio needs a parallel switchAudioQuality behavior to consume it.
• Channels-aware selection (5.1 / surround) — covered by 5.1-surround-selection. Audio tracks carry a channels field surfaced by the parser, but it's not yet used for selection or capability filtering.
• Audio-only composition optimizations — the engine tolerates audio-only sources today (basic coverage via engine.test.ts "handles audio-only stream"), but the default engine doesn't explicitly compose for audio-only contexts (no shorter buffer targets, no subtracted video-related work). The explicit variant ships via the audio-only-mode-override use case (Phase 1 landed); detect-from-parser routing in the default engine to supplant the tolerance is future work tracked there.
• A/V sync handling — delegated entirely to the browser's MSE pipeline. The engine doesn't intervene on audio/video sync.

Implementation surface

Composition: packages/spf/src/playback/engines/hls/engine.ts — audio behaviors composed alongside video and text:

// Track selection (reads config for initial preferences).
selectAudioTrack,
selectTextTrack,

// Resolve selected tracks (fetch media playlists)
resolveVideoTrack,
resolveAudioTrack,
resolveTextTrack,

// ...

// MSE setup
// ...
setupAudioBufferActors,

// Segment loading
// ...
loadAudioSegments,

Behaviors:

Behavior	File	Responsibility
selectAudioTrack	packages/spf/src/playback/behaviors/select-tracks.ts	Default audio rendition selection on source load. Lifecycle-only; mutually exclusive with switchAudioTrack
switchAudioTrack	packages/spf/src/playback/behaviors/dom/switch-audio-track.ts	Filter-reactive slot owner + mid-stream flush dispatcher. Owned architecturally by multi-language-audio — composed in both createSimpleHlsEngine and createHlsAudioOnlyEngine today
resolveAudioTrack	packages/spf/src/playback/behaviors/resolve-track.ts	Fetches the selected audio media playlist
setupAudioBufferActors	packages/spf/src/playback/behaviors/dom/setup-buffer-actors.ts	Audio SourceBuffer + actor setup. Owned architecturally by mse-mms-pipeline
loadAudioSegments	packages/spf/src/playback/behaviors/dom/load-segments.ts	Audio segment loading dispatcher. Owned architecturally by buffer-management

Helpers:

Helper	File	Status
pickAudioTrack(presentation, config)	packages/spf/src/media/primitives/select-tracks.ts	Default picker today — language-aware three-tier (preferredAudioLanguage → DEFAULT=YES → first-track). Wired by multi-language-audio Tier 1
pickFirstTrackId(presentation, 'audio')	packages/spf/src/media/primitives/select-tracks.ts	Simple first-track fallback (still available for callers overriding via SelectAudioTrackConfig.picker)

State slots:

• selectedAudioTrackId — single-writer. Owner depends on which audio- selection behavior is composed: selectAudioTrack (lifecycle-only) or switchAudioTrack (filter-reactive + mid-stream flush; the variant composed in createSimpleHlsEngine and createHlsAudioOnlyEngine today). Becomes switchAudioQuality's responsibility when audio-abr lands (extends switchAudioTrack).
• userAudioTrackSelection — added by multi-language-audio. Consumer-driven Partial<AudioTrack> filter narrowing the audio candidate set before switchAudioTrack's picker runs.
• Reads presentation (audio renditions surface in presentation.selectionSets)

Manifest parsing: parseMultivariantPlaylist (packages/spf/src/media/hls/parse-multivariant.ts) extracts audio renditions from #EXT-X-MEDIA:TYPE=AUDIO lines.

Config surface

{
  preferredAudioLanguage?: string;   // Exposed but inert with the
                                      // default picker; see What's
                                      // not implemented
}

Plus the behavior-level override:

// SelectAudioTrackConfig (consumed via composition config)
{
  picker?: TrackPicker<SelectAudioTrackConfig>;  // Override default
                                                  // pickFirstTrackId
}

Consumers wanting language-aware selection today must override picker — e.g., to use pickAudioTrack from @videojs/spf primitives.

Verification

• Unit tests:
  • packages/spf/src/playback/engines/hls/tests/engine.test.ts → "orchestrates complete pipeline" — asserts selectedAudioTrackId defined, audio track resolved, audio buffer actor present
  • packages/spf/src/playback/engines/hls/tests/engine.test.ts → "handles audio-only stream" — basic audio-only coverage
  • packages/spf/src/playback/behaviors/dom/tests/setup-buffer-actors.test.ts — per-type audio setup
  • Coverage gap: no dedicated select-tracks.test.ts coverage specifically asserting audio picker behavior (the default pickFirstTrackId path is exercised through the orchestration tests; the pickAudioTrack primitive has its own unit tests in media/primitives/tests/)
• Sandbox:
  • apps/sandbox/src/spf-segment-loading/ — exercises audio playback as part of HLS playback

Open questions

• Audio-only composition guarantees. The engine tolerates audio-only sources today, but is it designed for them? The audio-only-mode-override use case (Phase 1 landed) supplies the explicit variant; whether the default engine routes to it for audio-only sources via detect-from-parser is future work tracked there.
• Channels exposure vs use. The parser surfaces channels on audio tracks but no selection logic consumes it. Stays inert until 5.1-surround-selection wires it.

Related features

• multi-language-audio (partial, sketched) — the extension covering multi-rendition surfacing (free via parser), language-aware default selection (wires pickAudioTrack), programmatic selection via userAudioTrackSelection filter, and same-codec mid-stream switching via next-segment-boundary flush. Today's audio-playback is the baseline it builds on.
• subtitles — parallel structure (default picker + per-type segment loading; same setupTrackResolution helper). Text-track selection is user opt-in by default; audio's default selection always picks something (or nothing if no audio renditions).
• video-abr — video equivalent. Video has ABR overlay; audio doesn't (yet — see audio-abr).
• mse-mms-pipeline — owns audio SourceBuffer setup (setupAudioBufferActors). The Firefox mozHasAudio cross-type invariant lives in that feature's documentation.
• buffer-management — owns audio segment loading (loadAudioSegments). Same 4-state FSM as video / text.
• source-replacement — audio actors and selectedAudioTrackId tear down via the resolved/unresolved cascade.
• preload-modes — gates audio loading via the same FSM that gates video and text.
• audio-abr — bandwidth sampling + quality switching for audio. Parallel sibling of video-abr on the audio axis; consumes audio-playback's single-rendition baseline + (when it lands) multi-language-audio's rendition-group machinery.
• 5.1-surround-selection — channels-aware codec-change selection. Consumer of capability- probing on the audio channel-count axis. Wires the channels field surfaced by the parser into selection logic.
• audio-only-mode-override (use case; Phase 1 landed) — engine variant optimized for audio-only delivery; covers both truly-audio-only sources and mixed sources delivered as audio-only.
• capability-probing (candidate) — narrows the audio candidate set selection runs over; unsupported codecs filtered upstream of the selectAudioTrack picker.

Use cases that compose this feature

• audio-only-mode-override (partial — Phase 1 landed) — Phase 1 baseline constituent. The audio-only delivery variant composes this feature's rendition selection, media playlist resolution, and segment loading as-is via createHlsAudioOnlyEngine.

See also

• presentation-modeling.md — architectural deep-dive on the format-neutral data shape and parser interface that surfaces audio renditions (the parsePresentation contract this feature's recognition phase relies on)
• multi-language-audio.md — future extension on top of this feature
• clusters.md § Track & variant registry
• conventions/behaviors.md — per-type specialization pattern (setupTrackResolution)
• packages/spf/docs/hls-engine.md — engine composition walkthrough (Stage 2: track selection; Stage 3: track resolution)