mirror of https://github.com/zoriya/v10.git synced 2026-08-16 02:45:09 +00:00

Files

T

057f32573e feat(spf): multi-track audio + skills building features and behaviors (#1605 )

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-06-04 09:53:13 -07:00

23 KiB

Raw Blame History

status, date, definition

status	date	definition
partial	2026-05-21	sketched

Audio-only mode override

Engine variant that delivers audio-only playback. Composes regardless of source shape: handles both truly audio-only HLS sources (manifests with no video stream-inf entries) and mixed-AV sources (video / subtitle renditions ignored at composition time). The variant decision is encoded in the adapter choice — instantiating SimpleHlsAudioOnlyMediaElement opts the consumer into audio-only delivery.

This is a Player-level composition variant per ../features/clusters.md § Feature classification axes. It subsumes what Notion originally framed as two separate concerns (Notion epic #4a "Basic Audio-only" — source-shape correctness — and epic #4b "Audio-only Mode Override" — delivery-mode choice) into a single composition with two variant-decision-source paths (see Variant-decision signal source below).

Status

Composition: Phase 1 implemented. createHlsAudioOnlyEngine and SimpleHlsAudioOnlyMediaElement ship in packages/spf/src/playback/engines/hls/ and re-export from @videojs/spf/hls. The adapter handles both truly audio-only HLS sources and mixed-AV sources (video / subtitle renditions ignored at composition time). Phases 2 (audio-abr, multi-language-audio) and 3 (alternative buffer-target defaults) are not yet implemented.
Definition depth: sketched — Phase 1 implementation surface populated; Phases 2 and 3 remain coarse pending their constituent features.
Source material: Notion epics #4a (Basic Audio-only, Eng=S / Validation=S) and #4b (Audio-only Mode Override, Eng=M / Validation=S). Originally classified as separate Case-1 (Media-src composition case) + Case-2 (Player composition case) concerns; consolidated here because both ship the same engine factory and adapter, distinguished only by which signal sources the variant decision.

Target delivery context

Source-shape correctness — truly audio-only sources (where the manifest declares only audio renditions):

Audio-only HLS assets — sources published as audio-only (podcasts, music, audio articles). The current default createSimpleHlsEngine tolerates these via setupVideoBufferActors / loadVideoSegments no-op-ing when presentation.videoTracks is empty; this variant composes the audio-only pipeline explicitly, saving the no-op overhead and making the "no video" state structural rather than incidental.

Delivery-mode choice — mixed-AV sources delivered as audio-only:

"Listen on the go" toggles — user opts into audio-only delivery to save data or because they're not viewing the screen (background playback, screen off, headphones-only).
Audio-podcast modes — content is podcast-flavored even when the source manifest is a standard mixed-AV HLS asset (no audio-only-source variant published).
Mux-billing-driven audio-only sessions — Mux billing context may make audio-only sessions cheaper; consumers opt into audio-only to reduce billing.
Adapter-layer audio-only player chrome — consumer surface presents an audio-only UI (cover art, podcast controls), engine variant matches the surface.

Common to all: the delivery is audio-only. The variant composes identically across both source shapes; what differs is the variant-decision source (see below).

Phases of complexity

Phase	What
1 — Basic functionality (implemented)	Parallel engine-factory + adapter pair. `createHlsAudioOnlyEngine` composes the audio-side subset of `createSimpleHlsEngine`'s behavior list, subtracting video-side and text-track behaviors entirely (Phase 1 ships without subtitle support). `SimpleHlsAudioOnlyMediaElement` wraps the engine with the same `shareSignals`-based pattern as `SimpleHlsMediaElement`. See Implementation surface below
2 — Features/functionality relevant to the use case	Compose constituent feature behaviors as they land: `audio-abr` when implemented (multi-bitrate audio support in the variant), `multi-language-audio` when implemented (language selection within the variant). Both are additive — the variant gains capability as the constituent features get built
3 — Optimizations	Alternative default configurations for the audio-only delivery context: shorter forward-buffer targets (audio is lower-bandwidth; less ahead-buffering needed), possibly different `preload` defaults. The Path-A (update existing behavior's defaults) vs Path-B (audio-only-specific buffer-management behavior) judgment call applies — see `README.md` § Implementation note

Composition specifics

Phase 1 is subtractive-only; Phases 2 and 3 surface the other mechanisms.

Behaviors subtracted (Phase 1)

From createSimpleHlsEngine's composition, omit:

Video-side:

resolveVideoTrack — no video media playlist fetch
switchVideoQuality — no video ABR switching (also subsumes the default-pick that owns selectedVideoTrackId; see engine.ts:240-243)
setupVideoBufferActors — no video SourceBuffer / actor
loadVideoSegments — no video segment loading

Text-track-side (Phase 1 ships without subtitle support; future phase may add back):

selectTextTrack — no subtitle rendition selection
resolveTextTrack — no subtitle media playlist fetch
syncTextTracks — no TextTrack slot sync
setupTextTrackActors — no text-track actors
loadTextTrackSegments — no subtitle segment loading

initialState seed dropped: the parent engine seeds bandwidthState to let switchVideoQuality fire on its initial subscribe. With switchVideoQuality subtracted and no Phase 1 audio behavior subscribing to bandwidth, the seed becomes unnecessary and is omitted. Returns when audio-abr (Phase 2) lands.

Behaviors added (Phase 1)

None. This use case ships as an independent adapter paired with its own engine factory. The variant-decision is encoded in the adapter choice itself — consumers instantiate SimpleHlsAudioOnlyMediaElement to opt in. No runtime variant-decision behavior is needed.

Alternative implementations (Phase 3 candidates)

Possibly an audio-only-tuned buffer-management behavior with different defaults (Path-B). Phase 3 surfaces this candidate; the Path-A vs Path-B judgment per README.md § Implementation note defers.

Alternative default configurations (Phase 3 candidates)

forwardBuffer.bufferDuration — shorter for audio-only (e.g., 10s instead of 30s) since audio is lower-bandwidth and there's no display-paced render to ahead-buffer for.
preload default — potentially more eager for audio-only (the cost of speculative fetching is lower).

Constituent features

Phase 1 baseline:

audio-playback — used as-is. Provides rendition selection, media playlist resolution, segment loading; the variant inherits all of it.
engine-adapter-integration — used with an alternative adapter shape. The variant ships its own SimpleHlsAudioOnlyMediaElement-style adapter parallel to SimpleHlsMediaElement. The shareSignals mechanism + mixin pattern compose unchanged; the consumer-facing API differs.
mse-mms-pipeline — used as-is. MediaSource + endOfStream gate compose unchanged across variants (per the uniform-across-tracks discipline — endOfStream reads mediaSource.sourceBuffers aggregately).
buffer-management — used as-is in Phase 1; Phase 3 surfaces alternative defaults.

Phase 2:

audio-abr (not yet implemented) — when implemented, composed in for multi-bitrate audio. Used as-is.
multi-language-audio (partial — landed) — composed in unchanged; variant state exposes userAudioTrackSelection slot. Variant engine composes switchAudioTrack (slot owner with filter + flush) instead of selectAudioTrack; setupAudioBufferActors carries over from the default engine. Consumer can write language filters or specific track IDs for mid-stream switching within the audio-only variant.

Customer-policy surface

The variant ships as an independent adapter parallel to SimpleHlsMediaElement:

// Default (mixed AV delivery)
const player = new SimpleHlsMediaElement();
player.src = mixedSourceUrl;

// Audio-only mode override
const audioPlayer = new SimpleHlsAudioOnlyMediaElement();
audioPlayer.src = sameMixedSourceUrl;

The adapter encodes the variant choice. Detect-from-parser routing in the default adapter (Variant-decision path 2) is not yet built — consumers currently select the adapter that matches their delivery surface explicitly.

Specific consumer-facing properties (loop flag, autoplay-muted defaults, etc.) are adapter-layer concerns, not engine-variant concerns. The engine-variant surface is the composition itself; the adapter wraps it with the API consumers actually call.

Variant-decision signal source

The variant composes identically regardless of signal source. Two paths exist; both target the same createHlsAudioOnlyEngine factory:

1. Adapter-upfront (implemented; Phase 1). Selecting SimpleHlsAudioOnlyMediaElement over SimpleHlsMediaElement is the variant choice. No detect-from-parser logic, no runtime config branch. Used by consumers that know they want audio-only delivery (the delivery-mode-choice scenarios in Target delivery context).

2. Detect-from-parser (future). A routing-from-default-adapter path where SimpleHlsMediaElement (or a higher-level adapter) detects an audio-only source shape from the parsed presentation (presentation.videoTracks empty) and switches its internal engine factory to createHlsAudioOnlyEngine for that source. Targets the source-shape-correctness scenario without forcing consumers of audio-only sources to opt into a separate adapter type. The default adapter would default to createSimpleHlsEngine for mixed sources and createHlsAudioOnlyEngine for audio-only ones; existing tolerance behavior would be supplanted by explicit composition. Not yet built.

Both paths can coexist. The shared factory is the load-bearing artifact; how the variant is signaled is orthogonal.

Likely cross-cutting impact

Adapter shape proliferation. Each use-case composition that gets its own adapter multiplies the adapter surface. As more use cases land (background-looping-video, video-only-mode-override, audio-podcast mode, etc.), the inventory of SimpleXHlsMediaElement classes grows. Worth flagging as a registry-level concern; may surface a future "use-case adapter factory" pattern.
Default-adapter routing change (when Variant-decision path 2 lands). Detect-from-parser routing in SimpleHlsMediaElement would change the default engine for audio-only sources from createSimpleHlsEngine (current tolerance) to createHlsAudioOnlyEngine (explicit composition). Compatible in observable behavior — both produce working audio playback — but the composition shape changes structurally. Worth a test-fixture pass when the routing lands.
audio-only endOfStream simplification. endOfStream reads mediaSource.sourceBuffers aggregately and composes unchanged across variants. With only one buffer in the audio-only variant, the gate naturally simplifies — no per-type changes needed (verified in Phase 1 integration tests).
Audio-only ABR composition (Phase 2). When audio-abr lands, the variant gains multi-bitrate audio support additively. The audio path's createTrackedFetch integration lives in audio-abr's implementation surface; this variant composes it in unchanged.

Open questions

Buffer-target tuning baseline. Phase 3 alternative defaults — empirical question of whether the default 30s forward-buffer is appropriate for audio-only or wants tuning. Defer until usage signals appear from Phase 1.
Mux-billing integration. If audio-only sessions are billed differently, where does the billing context surface? Adapter-layer (consumer reports session type) vs engine-layer (engine exposes "this is audio-only mode" signal). Likely adapter-layer.
Variant-decision source for live + audio-only. Live + audio-only intersection (radio-stream-like delivery from mixed source) — same adapter-upfront pattern, or does live add complications?
Subtitle support for Phase 2+. Phase 1 ships without subtitles. Some audio-only consumer scenarios (podcast transcripts, accessibility) want subtitles back; a future phase can add the text-track behaviors as an additive composition. Whether subtitles compose unchanged from the default engine or need a tuning pass is open.

Resolved during Phase 1 implementation

These questions were open in the original coarse draft and resolved by the Phase 1 implementation pass (kept for traceability):

Engine variant factory shape — single shared factory. Originally the Case-1 source-shape concern and the Case-2 delivery-mode concern were documented as separate features+use-case (audio-only-composition + audio-only-mode-override); the Phase 1 implementation pass landed a single createHlsAudioOnlyEngine serving both, and the two docs consolidated into this one. Variant-decision source remains the orthogonal axis (see Variant-decision signal source above).
Adapter naming — SimpleHlsAudioOnlyMediaElement (with SimpleHlsAudioOnlyMediaMixin for mixin consumers and simpleHlsAudioOnlyMediaDefaultProps for default values), matching the Simple{Variant}HlsMediaElement naming pattern.
Subtitle / text-track handling for Phase 1 — subtracted entirely. Phase 1 ships without subtitles to keep the initial surface lean; future phases may add back additively (see Open questions).

Implementation surface

Phase 1 landed across the full SPF→core→html/react→sandbox cascade, parallel to the existing simple-hls-video pair.

SPF — engine factory (packages/spf/src/playback/engines/hls/):

Export	File	Purpose
`createHlsAudioOnlyEngine`	`engine-audio-only.ts`	Subtractive composition variant; omits all video + text-track behaviors
`SimpleHlsAudioOnlyEngineState`	`engine-audio-only.ts`	Trimmed state — no `selectedVideoTrackId` / `selectedTextTrackId` / `userVideoTrackSelection` / `bandwidthState`
`SimpleHlsAudioOnlyEngineContext`	`engine-audio-only.ts`	Trimmed context — no video buffer / video segment loader / text-track actor slots
`SimpleHlsAudioOnlyEngineConfig`	`engine-audio-only.ts`	Trimmed config — no video-quality, bandwidth, or text-track fields
`SimpleHlsAudioOnlyEngineSignals`	`engine-audio-only.ts`	`onSignalsReady` callback type

SPF — adapter (packages/spf/src/playback/engines/hls/):

Export	File	Purpose
`SimpleHlsAudioOnlyMediaElement`	`adapter-audio-only.ts`	Standalone adapter, no base class
`SimpleHlsAudioOnlyMediaMixin`	`adapter-audio-only.ts`	Mixin for adapters with a custom base
`SimpleHlsAudioOnlyMediaProps` / `…API`	`adapter-audio-only.ts`	Adapter API surface — same WHATWG src/preload/play contract as `SimpleHlsMediaElement`
`simpleHlsAudioOnlyMediaDefaultProps`	`adapter-audio-only.ts`	Default-prop constants

Public re-export: @videojs/spf/hls — all of the above ship via packages/spf/src/playback/engines/hls/index.ts.

Core — media wrapper (packages/core/src/dom/media/simple-hls-audio-only/):

Export	File	Purpose
`SimpleHlsAudioOnlyMedia`	`index.ts`	Applies `SimpleHlsAudioOnlyMediaMixin` to `HTMLAudioElementHost` (audio host, not video — symmetric with `mux-audio`)

Public re-export: @videojs/core/dom/media/simple-hls-audio-only.

HTML — custom element (packages/html/src/media/simple-hls-audio-only/):

Export	File	Purpose
`SimpleHlsAudioOnly`	`media/simple-hls-audio-only/index.ts`	Applies `MediaAttachMixin` + `CustomMediaElement('audio', SimpleHlsAudioOnlyMedia)`
`SimpleHlsAudioOnlyElement` (tag `simple-hls-audio-only`)	`define/media/simple-hls-audio-only.ts`	Custom-element definition; registers `<simple-hls-audio-only>` via `safeDefine`

CDN entry: packages/html/src/cdn/media/simple-hls-audio-only.ts → @videojs/html/cdn/media/simple-hls-audio-only.

React — component (packages/react/src/media/simple-hls-audio-only/):

Export	File	Purpose
`SimpleHlsAudioOnly`	`index.tsx`	React component rendering an `<audio>` element bound to `SimpleHlsAudioOnlyMedia`; props mirror the HTML element's WHATWG-style attribute surface

Public re-export: @videojs/react/media/simple-hls-audio-only.

Composed behaviors (current): syncPreload, trackLoadTriggers, resolvePresentation, resolveAudioTrack, calculatePresentationDuration, setupMediaSource, updateMediaSourceDuration, setupAudioBufferActors, trackCurrentTime, switchAudioTrack, loadAudioSegments, endOfStream, shareSignals. (Phase 1 composed selectAudioTrack; switchAudioTrack replaced it when multi-language-audio Tier 2 landed.)

endOfStream composes unchanged from the default engine — it iterates buffer actors via [videoBufferActor, audioBufferActor].filter(Boolean) and reads mediaSource.sourceBuffers aggregately, so the audio-only configuration drives end-of-stream correctly with no per-type changes.

Verification

Unit tests (packages/spf/src/playback/engines/hls/tests/):

engine-audio-only.test.ts:
- "creates engine with state, context, and destroy" — basic API surface
- "does not seed bandwidthState (no ABR behavior subscribed at init)" — asserts the dropped seed
- "plays truly audio-only HLS source (parity with default-engine tolerance)" — asserts selectedAudioTrackId, audioBufferActor, and an open MediaSource against an audio-only manifest. Parity baseline with engine.test.ts → "handles audio-only stream (no video tracks)"
- "plays mixed HLS source as audio-only (video tracks ignored)" — the Case-2 acceptance criterion. Asserts the variant ignores video stream-inf entries: no selectedVideoTrackId, no videoBufferActor, no fetch of the video media playlist
- "plays mixed HLS source with subtitles ignored" — asserts no text-track machinery is set up against a manifest that contains a TYPE=SUBTITLES rendition
- "cleans up on destroy" — engine.destroy() does not throw
adapter-audio-only.test.ts — 28 tests covering the WHATWG src/preload/play()/attach/detach/destroy contract on SimpleHlsAudioOnlyMediaElement. Mirrors adapter.test.ts semantically (the variant differs in composition, not in adapter contract).

Sandbox demos (templates checked in; src/ is mirrored from templates/ by pnpm dev:sandbox per the sandbox README):

apps/sandbox/templates/html-simple-hls-audio-only/ — exercises <simple-hls-audio-only> against the shared SOURCES registry, wrapped in the standard audio-player skin shell. Sandbox parity with html-mux-audio / html-simple-hls-video.
apps/sandbox/templates/react-simple-hls-audio-only/ — exercises <SimpleHlsAudioOnly> against the same registry through the React audio skin component. Sandbox parity with react-mux-audio / react-simple-hls-video.

Out of scope for Phase 1 (deferred):

E2E tests — engine-level integration + manual sandbox verification suffice for Phase 1; E2E reliability coverage deferred until behavior stabilizes.

audio-playback — constituent baseline.
engine-adapter-integration — constituent; variant adapter parallels SimpleHlsMediaElement.
mse-mms-pipeline — constituent; composes unchanged.
buffer-management — constituent; Phase 3 alternative defaults candidate.
audio-abr (documented; pending implementation) — Phase 2 constituent.
multi-language-audio (coarse; not yet implemented) — Phase 2 constituent.

video-only-mode-override (coarse) — inverse-axis sibling. Same shape, video-side instead of audio-side: subtract audio behaviors from any source to deliver video-only. Distinct from [background-looping-video] despite shared constituent features.

23 KiB Raw Blame History