diff --git a/internal/design/spf/features/capability-probing.md b/internal/design/spf/features/capability-probing.md index e22840de..a7842d88 100644 --- a/internal/design/spf/features/capability-probing.md +++ b/internal/design/spf/features/capability-probing.md @@ -1,7 +1,7 @@ --- -status: draft +status: partial date: 2026-05-20 -definition: technical +definition: sketched --- # Capability probing @@ -25,18 +25,19 @@ DRM, and the unsupported-case error mapping. ## Status -- **Composition:** not implemented in `createSimpleHlsEngine`. A - single late-failure check exists today: `isCodecSupported` in - `media/dom/mse/mediasource-setup.ts` is called by `createSourceBuffer` - inside `setupVideoBufferActors` / `setupAudioBufferActors` — by which - point selection has already run and an unsupported variant may have - been picked. -- **Definition depth:** technical — scope and constraints articulated; - no implementation. Source material: [SPF Epics Working Doc — - candidate epics #17 (codec / container), #18 (multivariant CODECS), - #19 (key-system)](https://www.notion.so/35f97a7f89d08123a13fecab1ca1cac4) - (cluster D; each sized S; the Epics doc flags #17 + #18 as likely - merging). +- **Composition:** synchronous codec filtering is live in + `createSimpleHlsEngine`. The `canPlayTrack` probe is injected into + `track-switching`'s hard-constraints pre-pass (`excludeUnplayableTracks`), + so undecodable renditions are pruned *before* selection. The pre-existing + late-failure check (`isCodecSupported` inside `createSourceBuffer`) stays as + a defensive backstop and should now rarely fire. +- **Definition depth:** sketched — Phase 1 (probe primitive) and Phase 2 + (multivariant CODECS filtering) are implemented; key-system probing, + `changeType()` probing, segment-level checking, no-playable surfacing + (Phase 6), and Tier 2 overrides remain unimplemented. Source + material: [SPF Epics Working Doc — candidate epics #17 (codec / container), + #18 (multivariant CODECS), + #19 (key-system)](https://www.notion.so/35f97a7f89d08123a13fecab1ca1cac4). - **Foundational** for cluster D — `[hevc-variant-selection]`, `[5.1-surround-selection]`, `[unsupported-case-error-mapping]`, and [drm-support](./drm-support.md) (GitHub issue #1411) all consume @@ -49,15 +50,15 @@ Scope slices around the capability-probing contract. Tier 1 layer onto specific phases per the [Tier 1 / Tier 2 framing](./clusters.md#tier-1-spec-compliant-baseline-vs-tier-2-custom-behavior). -| Phase | What | Notes | +| Phase | What | Status | |---|---|---| -| Codec / container probing primitive | Uniform API wrapping `MediaSource.isTypeSupported` + `canPlayType`. Helpers for "given a `Track`, can we play it?" Builds on today's `isCodecSupported` | The minimum primitive everything else builds on. Today's `isCodecSupported` is the codec half; the wrapper formalizes the surface | -| Multivariant CODECS-attribute filtering | At presentation resolution (post-parse), filter `presentation.selectionSets` to drop renditions whose `CODECS` doesn't decode on this browser. Filtered set is what selection behaviors operate over; unsupported renditions never reach selection | Tier 1 (spec-compliant). Today's late-failure path becomes a defensive fallback rarely exercised | -| Media-playlist / segment-level capability checking | Per-segment CODECS verification + container detection at the media-playlist level. Catches mismatches the multivariant didn't declare | Tier 1. Largely defensive; expected to be rare for well-formed manifests | -| Key-system capability probing | `requestMediaKeySystemAccess` for each candidate key system (Widevine, PlayReady, FairPlay, FairPlay-AirPlay). Returns supported configurations. **DRM-adjacent boundary:** this feature owns Tier 1 probing only; EME setup, license fetch, key delivery live under [drm-support](./drm-support.md) (GitHub issue #1411) | Async — pushes toward a new-behavior filter writer pattern rather than a derived signal | -| Cross-codec transition (`changeType()`) probing | Probe whether `SourceBuffer.changeType()` is available, plus pair-wise support for specific codec transitions (AVC ↔ HEVC, AAC stereo ↔ AC-3 5.1, etc.). Browser support is fragile and pair-specific | Consumers decide whether to attempt mid-stream switches based on this probe; the `changeType()` call itself lives in those consumer features | -| Unsupported-case error surfacing | When no candidate survives filtering, surface a clear error rather than failing late in `createSourceBuffer`. State-error slot or callback — the interface is defined here; consumer-side mapping lives in `[unsupported-case-error-mapping]` | The "fail loudly upstream" path | -| Tier 2: customer probing overrides | Config-driven biases: "force AVC even when HEVC supported," "prefer hardware-backed DRM," "exclude codec X." Layered on top of Tier 1's spec-compliant filtering | Tier 2 (custom behavior). Often consumer-policy-driven | +| Codec / container probing primitive | Uniform API wrapping `MediaSource.isTypeSupported` + `canPlayType`. Helpers for "given a `Track`, can we play it?" Builds on today's `isCodecSupported` | **Implemented** (codec half) — `canPlayTrack` in `media/dom/capabilities.ts`, memoized by MIME. `canPlayType` not wrapped yet | +| Multivariant CODECS-attribute filtering | Post-parse, drop renditions whose `CODECS` doesn't decode on this browser, before selection. Filtered set is what selection behaviors operate over | **Implemented** — `excludeUnplayableTracks` constraint in `track-switching`'s pre-pass. Tier 1 (spec-compliant). The late-failure path is now a defensive fallback | +| Media-playlist / segment-level capability checking | Per-segment CODECS verification + container detection at the media-playlist level. Catches mismatches the multivariant didn't declare | Not implemented. Tier 1; largely defensive, rare for well-formed manifests | +| Key-system capability probing | `requestMediaKeySystemAccess` for each candidate key system (Widevine, PlayReady, FairPlay, FairPlay-AirPlay). Returns supported configurations. **DRM-adjacent boundary:** this feature owns Tier 1 probing only; EME setup, license fetch, key delivery live under [drm-support](./drm-support.md) (GitHub issue #1411) | Not implemented. Async — a slot-writer behavior, *not* the synchronous config-predicate route codec filtering took (see resolved open question) | +| Cross-codec transition (`changeType()`) probing | Probe whether `SourceBuffer.changeType()` is available, plus pair-wise support for specific codec transitions (AVC ↔ HEVC, AAC stereo ↔ AC-3 5.1, etc.). Browser support is fragile and pair-specific | Not implemented. Consumers decide whether to attempt mid-stream switches based on this probe; the `changeType()` call itself lives in those consumer features | +| Unsupported-case error surfacing | When no candidate survives filtering, surface a clear state rather than failing late in `createSourceBuffer` | Not implemented. A speculative per-type `noPlayable{Video,Audio}Tracks` flag was prototyped and **removed** (no consumer — write-only state + derivation stored as a slot). Today, when constraints prune a type that *has* tracks to empty, the behavior clears the selection (no pick) and `console.error`s as a placeholder; the late `createSourceBuffer` check is the backstop. Deferred in full to `[unsupported-case-error-mapping]`, which will replace the `console.error` with a surfaced shape (likely a derived `computed`, not a stored slot) once a consumer exists | +| Tier 2: customer probing overrides | Config-driven biases: "force AVC even when HEVC supported," "prefer hardware-backed DRM," "exclude codec X." Layered on top of Tier 1's spec-compliant filtering | Not implemented. The `canPlayTrack` config injection point is the natural seam (override the default probe) | ## What's in scope vs out of scope @@ -90,6 +91,42 @@ layer onto specific phases per the - Adapter / consumer-side error display - Consumer-specific error-code mappings (above-engine) +## Implementation surface + +**Probe primitive:** + +| Export | File | Role | +|---|---|---| +| `canPlayTrack` | `media/dom/capabilities.ts` | `(track) => boolean` — builds the MIME codec string and checks `MediaSource.isTypeSupported`, memoized by MIME. Unprobeable tracks (no `mimeType` / no `codecs`) pass through. **Asserts detected non-fMP4 containers (`video/mp2t`, `audio/aac`) unsupported without probing** — TS because the probe false-positives + no transmux; raw AAC because the pipeline assumes init segments (a temporary limitation — browser supports it) | +| `CanPlayTrack` (type) | `media/types/index.ts` | DOM-free predicate type the constraint consumes; `canPlayTrack` is its DOM implementation | +| Container detection (`CONTAINER_MIME_BY_EXTENSION`, `NON_FMP4_CONTAINER_MIMES`) | `media/hls/parse-media-playlist.ts` | Detects non-fMP4 containers per media playlist (no `#EXT-X-MAP` **and** a recognized segment extension: `.ts` → `video/mp2t`, `.aac` → `audio/aac`) and relabels the resolved track's `mimeType` from the fMP4 default. `canPlayTrack` then prunes them | +| `applyContainerMimeType` | `media/utils/tracks.ts` | Propagates the detected container to every rendition of the **same type** (called from `resolve-track`): one resolved non-fMP4 playlist relabels all of that type's renditions, so the type is pruned from a single fetch. Scoped to one type — never crosses audio↔video (mixed-container sources exist), which also keeps per-type resolutions' writes disjoint (race-free) | + +**Constraint (in `playback/behaviors/track-switching.ts`):** + +| Piece | Role | +|---|---| +| `excludeUnplayableTracks` | Hard-constraint in the `applyConstraints` pre-pass; reads `config.canPlayTrack`, drops undecodable renditions before the rule chain. Shared by `switchVideoTrack` / `switchAudioTrack`, pooled with `excludeFailedCdns`. When it prunes a type that *has* tracks to empty, the selection clears (no pick) so a pick made before a relabel can't linger as unplayable | + +**Engine wiring (`playback/engines/hls/engine.ts` + `engine-audio-only.ts`):** +- `canPlayTrack` config — both engine factories default it to the DOM `canPlayTrack` in `finalConfig` (the audio-only variant too, so filtering isn't inert there); override to force-exclude a codec (the Tier 2 seam). Adapters forward it via `...config`. + +**State slots:** +- **Reads (constraint):** `presentation` candidates' `mimeType` + `codecs`, via `config.canPlayTrack`. +- **Writes:** none — the constraint only narrows the candidate set the existing `selected*TrackId` writers pick from. + +## Verification + +- **Unit — `media/dom/tests/capabilities.test.ts`:** `canPlayTrack` returns the `isTypeSupported` verdict for a track's built MIME; memoizes per unique MIME (probes once); passes through (`true`) for unprobeable tracks (no `mimeType`, or empty/absent `codecs`); asserts non-fMP4 containers (`video/mp2t`, `audio/aac`) unsupported without consulting `isTypeSupported` (even with codecs). +- **Unit — `media/hls/tests/parse-media-playlist.test.ts`:** relabels to `video/mp2t` / `audio/aac` when there's no `#EXT-X-MAP` and segments are `.ts` / `.aac` (query string ignored; `video/mp2t` for audio TS too); keeps the fMP4 default when an `#EXT-X-MAP` is present or the extension is unrecognized. +- **Unit — `media/utils/tests/tracks.test.ts`:** `applyContainerMimeType` sets the MIME on every track of the given type, leaves other types untouched (never crosses audio↔video), idempotent. +- **Unit — `playback/behaviors/tests/track-switching.test.ts`:** `excludeUnplayableTracks` prunes undecodable renditions before ranking (picks best playable codec); passes through with no probe wired; a user-selected unplayable track is still excluded (hard constraint beats the soft user filter); makes no pick when the constraint prunes every rendition from a cold start; **clears a prior pick when a later relabel prunes every rendition to empty** (the warm path — a pick made under the initial mp4 label, then the type relabeled to `video/mp2t`). +- **Integration — `playback/engines/hls/tests/engine.test.ts`:** a mixed HEVC+AVC source with a `canPlayTrack` rejecting HEVC selects the AVC rendition; an all-undecodable source makes no video pick. + +**Live smoke test:** verified in the SPF sandbox against the Apple `bipbop_4x3` stream (muxed-TS video + raw-`.aac` audio) — video relabels `video/mp2t`, audio relabels `audio/aac`, and both types are pruned to empty (per-type propagation: one fetch per type, not per rendition), so the selection clears (no pick) instead of leaving the initial mp4-labeled pick selected to stall deep in the pipeline. An audio-only `.aac` source likewise makes no pick. + +**Out of scope / deferred:** `canPlayType` wrapper, key-system probing, `changeType()` probing, per-segment CODECS checking, no-playable surfacing / the full error-code interface, and Tier 2 override config. Container *detection* covers MPEG-TS + raw ADTS AAC (not `.mp3` etc. yet); both are asserted **unplayable** for now. *Playing* them is separate follow-up work — TS needs a transmux pipeline; **raw AAC is genuinely browser-supported (Chrome/Safari) and could be played by removing the pipeline's init-segment assumption** (the segment loader queues an `append-init` with an empty URL, and append handling is fMP4-shaped). + ## Likely cross-cutting impact - **Selection behaviors** — `selectVideoTrack` / `switchVideoQuality` @@ -115,30 +152,85 @@ layer onto specific phases per the `changeType()` call lives in their own feature work; this feature provides the "can we?" answer. +## Resolved during Phase 1–2 implementation + +- **Filter-writer pattern → split (not unified).** Synchronous codec + probing is a pure function of (track, environment), so it landed as a + **config-injected predicate** (`canPlayTrack`) read by the + `excludeUnplayableTracks` hard constraint — no behavior, no state slot. + This mirrors `getCdnId` injection, not the `failedCdns ← + setupFailoverMonitor` slot-writer (which exists because CDN cooldown is + *dynamic*; codec support is static). Async key-system probing (Phase 4) + will take the slot-writer route, coexisting in the same pre-pass — the + two are genuinely split by sync-vs-async. +- **Cache-eager vs lazy → lazy + memoized.** The constraint probes each + candidate at apply time; `canPlayTrack` memoizes by built MIME string, so + each unique MIME is asked once without an upfront sweep. +- **No-playable surfacing was removed (not deferred-as-built).** A per-type + `noPlayable*` flag was prototyped — set when a type's candidates pruned to + empty (codec filtering *or* CDN-failover cooldown). It was deleted: nothing + consumed it (write-only state), and it stored a derivation in a slot an + effect wrote rather than a `computed`. Today an emptied type that *had* tracks + clears the selection (no pick) and `console.error`s as a placeholder; the late + `createSourceBuffer` check is the backstop, and a track with no declared + `CODECS` (optional per spec) is unprobeable and passes through. The + *had-candidates-vs-never-had-candidates* distinction a bare "candidate set is + empty" check can't make (a video-only source's absent audio is empty but not + an error) is **already** made — the clear/error is gated on a non-zero + pre-constraint track count for the type. When a consumer materializes, + `[unsupported-case-error-mapping]` replaces the `console.error` with the + surfaced shape (likely a `computed`). +- **Container-detection scope → non-fMP4 detection (TS + raw AAC), per-track-type, + marked unplayable.** The media-playlist parser relabels a resolved non-fMP4 + rendition (no `#EXT-X-MAP` + a recognized extension: `.ts` → `video/mp2t`, + `.aac` → `audio/aac`), and `resolve-track` propagates that MIME to every + rendition of the **same type** (`applyContainerMimeType`), so a type is pruned + from a single resolved playlist. Propagation is **same-type, not cross-type**: + Apple `bipbop_4x3` is mixed-container (muxed-TS video + raw-`.aac` audio), so a + cross-type "whole source is one container" assumption was both *wrong* for that + stream and racy under concurrent per-type resolution; same-type is safe (an ABR + ladder shares its container) and race-free (disjoint writes). Both are asserted + unplayable, for different reasons: TS because `isTypeSupported('video/mp2t…')` + false-positives on Chromium *and* there's no transmux pipeline; raw AAC as a + **temporary** limitation — the browser genuinely supports it (Chrome/Safari + decode bare `audio/aac`; Firefox doesn't), but our segment loader / append + pipeline assumes an `EXT-X-MAP` init segment, so it would fetch-but-never-buffer + (a silent stall) today. Making raw AAC playable (remove the init-segment + assumption; bare-MIME probe + projection) is deliberately out of scope here — + see [container-support](./container-support.md). + ## Open questions -- **Filter-writer pattern.** New behavior vs derived signal? Async - key-system probing pushes toward new-behavior (writes a filter - slot after async resolves). Synchronous codec probing could be a - derived signal. Unified or split? -- **Cache-eager vs lazy probing.** Probe everything upfront (simpler, - worst-case cost) vs probe-on-demand (more efficient for sources - with many renditions). Affects state-slot writer pattern. +- **Surfacing "nothing playable" as a (fatal) error.** The pre-pass can prune a + type to empty — every rendition undecodable (fatal: the source can't play at + all) or, transiently, every CDN cooled down (recoverable). Today both clear + the selection (no pick) and `console.error` — *uniformly*, without yet + distinguishing fatal from recoverable; the late `createSourceBuffer` check + remains the backstop. `[unsupported-case-error-mapping]` will want this + surfaced. Open: a derived + `computed` read by that consumer (preferred — see the removed `noPlayable*` + note under "Resolved"), or a dedicated **behavior** if surfacing needs to own + emit/clear, the fatal-vs-recoverable classification, and the + had-vs-never-had-candidates gating? Lean `computed` unless that + lifecycle/coordination forces a behavior. Whatever lands must distinguish + fatal (no decodable rendition) from recoverable (failover cooldown) so the + consumer doesn't fire a terminal error on a transient condition. +- **Sharing the constraint pre-pass with user-selection logic.** Constraints + (`excludeUnplayableTracks`, `excludeFailedCdns`) live *inside* + `track-switching`'s pre-pass, invisible outside it. A hard constraint already + correctly beats a user pick (tested — the user's unplayable selection is + pruned before `filterByUserSelection` runs), but the override is *silent* and + a user-facing track list can still present a rendition that will never play. + Open: expose the post-constraint candidate set (per type) as shared state for + selection UIs / the error surface to consume, or factor the constraint list + into a reusable unit those surfaces apply themselves? Same "expose the + candidate set" thread as the `noPlayable*` removal. - **`changeType()` pair-wise probing API.** Probe all pairs upfront, probe lazily on switch attempt, or expose a `canChangeType(from, to)` predicate that callers invoke? - **Tier 2 customer override surface.** Config-driven (engine-wide) vs per-source vs both? Per-source is more flexible but harder to wire. -- **Container-detection scope.** [container-support](./container-support.md) - is documented as standalone (cluster-less; MSE doesn't accept - non-fMP4 containers per spec, so the concern is structurally - different from capability-probing's "what can the browser decode - in fMP4?" framing). Open: should this feature surface container- - format detection as part of multivariant filtering (filter out - MPEG-TS variants when no transmuxer is composed)? That would - resolve the scope intersection cleanly without building the - transmuxer. ## Related features @@ -161,12 +253,12 @@ layer onto specific phases per the consumer-facing error mapping on top of this feature's error primitive. - **[container-support](./container-support.md)** — standalone - feature (cluster-less); resolved per the doc's framing that MSE - doesn't accept non-fMP4 containers per spec, making the concern - fundamentally different from capability-probing's framing. - Container-detection-without-transmuxer remains a possible cross- - feature integration point (filter MPEG-TS variants when no - transmuxer is composed). + feature (cluster-less) for *playing* non-fMP4 containers. The + detection half landed here: MPEG-TS (`.ts` → `video/mp2t`) and raw + ADTS AAC (`.aac` → `audio/aac`) are detected and asserted unplayable. + Playing them lives there — TS needs a transmux pipeline; raw AAC just + needs the init-segment assumption removed from the segment loader / + append pipeline (the browser already supports it). ## See also diff --git a/packages/spf/src/media/dom/capabilities.ts b/packages/spf/src/media/dom/capabilities.ts new file mode 100644 index 00000000..c6f1f893 --- /dev/null +++ b/packages/spf/src/media/dom/capabilities.ts @@ -0,0 +1,66 @@ +/** + * Capability probing — the engine's foundation for asking the browser what it + * can actually decode before committing a rendition to the pipeline. + * + * Today this is the synchronous codec half: `canPlayTrack` answers "can this + * environment play this track?" by building the track's MIME codec string and + * passing it to `MediaSource.isTypeSupported` (via `isCodecSupported`). It's the + * DOM implementation of the DOM-free `CanPlayTrack` predicate the + * track-switching hard-constraint pre-pass consumes — injected through engine + * config so the (DOM-free) behavior never imports a DOM API directly. + * + * Results are memoized by built MIME string: codec support is a pure function + * of (codec, environment) and never changes after load, so probing is lazy + * (per candidate, at constraint-apply time) but each unique MIME is asked once. + * + * Future cluster-D phases (async `requestMediaKeySystemAccess` key-system + * probing, `SourceBuffer.changeType()` availability) extend this surface; the + * async ones land as a state-slot writer behavior rather than a config + * predicate, since their verdict resolves asynchronously. + */ + +import { NON_FMP4_CONTAINER_MIMES } from '../hls/parse-media-playlist'; +import type { CanPlayTrack } from '../types'; +import { buildMimeCodec, isCodecSupported } from './mse/mediasource-setup'; + +const codecSupportCache = new Map(); + +/** + * Whether the environment can decode `track`, by codec. Builds the track's + * MIME codec string and checks `MediaSource.isTypeSupported`, memoized by MIME. + * A track without enough to probe — no `mimeType`, or no declared `codecs` + * (CODECS is optional per the HLS spec) — is unprobeable and passes through as + * playable (`true`) rather than being dropped; the late `createSourceBuffer` + * check stays as the backstop for those. + * + * Detected non-fMP4 containers (`video/mp2t`, `audio/aac`) are asserted + * unsupported regardless of the probe, so they're pruned before selection + * (the type makes no pick) instead of failing/stalling deep in the pipeline. + * Two different reasons, neither UA-based: + * + * - **MPEG-TS** can't be played at all here: `isTypeSupported('video/mp2t…')` is + * a genuine false positive on Chromium (reports `true` but appends produce no + * buffered range), and this engine has no TS transmux pipeline. + * - **Raw ADTS AAC** is a *temporary* limitation. The browser genuinely + * supports it (Chrome/Safari decode `audio/aac`; Firefox doesn't), so it could + * be made playable — but our segment actors / loading behaviors / append + * pipeline assume every rendition has an `EXT-X-MAP` init segment (e.g. an + * `append-init` task with an empty URL, fMP4-shaped append handling). Until + * that init-segment assumption is removed, ADTS would fetch but never buffer + * (a silent stall), so we assert it unplayable for now. FOLLOW-UP: drop the + * init-required assumption in the pipeline and switch this to a bare-MIME + * probe (`buildMimeCodec` would project `audio/aac` with no codecs) so it + * plays where the browser supports it. + * + * Override via the engine's `canPlayTrack` config when those pipelines land. + */ +export const canPlayTrack: CanPlayTrack = (track) => { + if (track.mimeType && NON_FMP4_CONTAINER_MIMES.has(track.mimeType)) return false; + if (!track.mimeType || !track.codecs?.length) return true; + const mimeCodec = buildMimeCodec({ mimeType: track.mimeType, codecs: track.codecs }); + const cached = codecSupportCache.get(mimeCodec); + if (cached !== undefined) return cached; + const supported = isCodecSupported(mimeCodec); + codecSupportCache.set(mimeCodec, supported); + return supported; +}; diff --git a/packages/spf/src/media/dom/tests/capabilities.test.ts b/packages/spf/src/media/dom/tests/capabilities.test.ts new file mode 100644 index 00000000..e4166c69 --- /dev/null +++ b/packages/spf/src/media/dom/tests/capabilities.test.ts @@ -0,0 +1,57 @@ +import { afterEach, describe, expect, it, vi } from 'vitest'; +import { canPlayTrack } from '../capabilities'; + +describe('canPlayTrack', () => { + afterEach(() => { + vi.restoreAllMocks(); + }); + + it('returns the isTypeSupported verdict for the track built MIME', () => { + const spy = vi.spyOn(MediaSource, 'isTypeSupported'); + spy.mockReturnValueOnce(true).mockReturnValueOnce(false); + + expect(canPlayTrack({ mimeType: 'video/mp4', codecs: ['supported.1'] })).toBe(true); + expect(canPlayTrack({ mimeType: 'video/mp4', codecs: ['unsupported.1'] })).toBe(false); + + expect(spy).toHaveBeenNthCalledWith(1, 'video/mp4; codecs="supported.1"'); + expect(spy).toHaveBeenNthCalledWith(2, 'video/mp4; codecs="unsupported.1"'); + }); + + it('memoizes by built MIME string — probes each unique MIME once', () => { + const spy = vi.spyOn(MediaSource, 'isTypeSupported').mockReturnValue(true); + const codecs = ['memo.unique.codec']; + + canPlayTrack({ mimeType: 'video/mp4', codecs }); + canPlayTrack({ mimeType: 'video/mp4', codecs }); + canPlayTrack({ mimeType: 'video/mp4', codecs: [...codecs] }); + + const calls = spy.mock.calls.filter(([mime]) => mime === 'video/mp4; codecs="memo.unique.codec"'); + expect(calls).toHaveLength(1); + }); + + it('passes through (true) for an unprobeable track with no mimeType', () => { + const spy = vi.spyOn(MediaSource, 'isTypeSupported'); + expect(canPlayTrack({ codecs: ['avc1.42E01E'] })).toBe(true); + expect(spy).not.toHaveBeenCalled(); + }); + + it('passes through (true) when codecs is empty or absent (unprobeable, CODECS optional)', () => { + const spy = vi.spyOn(MediaSource, 'isTypeSupported'); + expect(canPlayTrack({ mimeType: 'video/mp4', codecs: [] })).toBe(true); + expect(canPlayTrack({ mimeType: 'video/mp4' })).toBe(true); + expect(spy).not.toHaveBeenCalled(); + }); + + it('asserts non-fMP4 containers (video/mp2t, audio/aac) unsupported without consulting isTypeSupported', () => { + // TS: the probe false-positives on Chromium + no transmux. Raw AAC: a + // temporary limitation (the browser supports it, but our pipeline assumes an + // init segment) — both are pruned before selection rather than stalling. + const spy = vi.spyOn(MediaSource, 'isTypeSupported').mockReturnValue(true); + expect(canPlayTrack({ mimeType: 'video/mp2t', codecs: ['avc1.640028'] })).toBe(false); + expect(canPlayTrack({ mimeType: 'audio/aac', codecs: ['mp4a.40.2'] })).toBe(false); + // Even without codecs (the usual pass-through case), they're still dropped. + expect(canPlayTrack({ mimeType: 'video/mp2t' })).toBe(false); + expect(canPlayTrack({ mimeType: 'audio/aac' })).toBe(false); + expect(spy).not.toHaveBeenCalled(); + }); +}); diff --git a/packages/spf/src/media/hls/parse-attributes.ts b/packages/spf/src/media/hls/parse-attributes.ts index 045dd87c..11c91992 100644 --- a/packages/spf/src/media/hls/parse-attributes.ts +++ b/packages/spf/src/media/hls/parse-attributes.ts @@ -59,6 +59,14 @@ export function parseFrameRate(value: string): FrameRate | undefined { return { frameRateNumerator: Math.round(fps) }; } +// Audio codec identifiers, matched case-insensitively against each CODECS +// entry's prefix. Beyond AAC (`mp4a.*`): Dolby (`ac-3`, `ec-3`, `ac-4`), Opus, +// FLAC (`fLaC`), DTS (`dts*`), ALAC, and Vorbis. Recognizing these is what lets +// capability probing filter undecodable audio renditions (e.g. an AC-3 5.1 +// track on a browser without AC-3) — an unrecognized codec parses empty and is +// treated as unprobeable, so it would never be pruned. +const AUDIO_CODEC_PREFIXES = ['mp4a.', 'ac-3', 'ec-3', 'ac-4', 'opus', 'flac', 'dts', 'alac', 'vorbis']; + /** * Parse CODECS attribute into separate video and audio codecs. */ @@ -67,9 +75,10 @@ export function parseCodecs(codecs: string): { video?: string; audio?: string } const result: { video?: string; audio?: string } = {}; for (const codec of parts) { + const lower = codec.toLowerCase(); if (codec.startsWith('avc1.') || codec.startsWith('hvc1.') || codec.startsWith('hev1.')) { result.video = codec; - } else if (codec.startsWith('mp4a.')) { + } else if (AUDIO_CODEC_PREFIXES.some((prefix) => lower.startsWith(prefix))) { result.audio = codec; } } diff --git a/packages/spf/src/media/hls/parse-media-playlist.ts b/packages/spf/src/media/hls/parse-media-playlist.ts index 831e5a5e..edeb57d1 100644 --- a/packages/spf/src/media/hls/parse-media-playlist.ts +++ b/packages/spf/src/media/hls/parse-media-playlist.ts @@ -11,6 +11,39 @@ import type { import { matchTag, parseByteRange, parseExtInfDuration } from './parse-attributes'; import { resolveUrl } from './resolve-url'; +/** MPEG-2 Transport Stream (IANA `video/MP2T`, lowercased for `isTypeSupported`). Video + audio TS — there is no `audio/mp2t`. */ +export const MPEG_TS_MIME = 'video/mp2t'; +/** Raw ADTS AAC packed-audio (HLS `.aac` segments; IANA `audio/aac`). */ +export const RAW_AAC_MIME = 'audio/aac'; + +// Non-fMP4 container MIMEs keyed by segment file extension. fMP4 (the MSE +// default) always carries an EXT-X-MAP init segment, so a media playlist with +// no init segment and one of these extensions is a non-fMP4 rendition, +// relabeled from the fMP4 default. Extend with `.mp3` → 'audio/mpeg' etc. +const CONTAINER_MIME_BY_EXTENSION: Record = { + '.ts': MPEG_TS_MIME, + '.aac': RAW_AAC_MIME, +}; + +/** The non-fMP4 container MIMEs the parser detects — all currently treated as unplayable. */ +export const NON_FMP4_CONTAINER_MIMES = new Set(Object.values(CONTAINER_MIME_BY_EXTENSION)); + +/** + * Non-fMP4 container MIME for a (resolved, absolute) segment URL, by file + * extension, ignoring the query string. `undefined` for fMP4 / unrecognized. + */ +function containerMimeFromSegment(url: string | undefined): string | undefined { + if (!url) return undefined; + let path: string; + try { + path = new URL(url).pathname.toLowerCase(); + } catch { + path = url.toLowerCase().split('?')[0] ?? ''; + } + const dot = path.lastIndexOf('.'); + return dot === -1 ? undefined : CONTAINER_MIME_BY_EXTENSION[path.slice(dot)]; +} + /** * Resolve unresolved track type to its resolved equivalent. */ @@ -134,10 +167,20 @@ export function parseMediaPlaylist( ? { url: initSegmentUrl, ...(initSegmentByteRange ? { byteRange: initSegmentByteRange } : {}) } : { url: '' }; + // Container detection: fMP4 always carries an EXT-X-MAP init segment, so its + // absence plus a recognized non-fMP4 segment extension (`.ts` → MPEG-TS, + // `.aac` → raw ADTS AAC) marks a non-fMP4 rendition (high-precision — never + // trips on fMP4, which mandates the map). Relabel from the fMP4 default + // `video/mp4` / `audio/mp4` to the container MIME so capability probing prunes + // it (these containers are currently treated as unplayable; see `canPlayTrack`). + const detectedContainer = initSegmentUrl ? undefined : containerMimeFromSegment(segments[0]?.url); + const mimeType = unresolved.type !== 'text' && detectedContainer ? detectedContainer : unresolved.mimeType; + // Generic resolution: All type-specific fields already on unresolved track from P1 // Just add parsed properties (startTime, duration, segments, initialization) return { ...unresolved, + mimeType, startTime: 0, duration: totalDuration, segments, diff --git a/packages/spf/src/media/hls/parse-multivariant.ts b/packages/spf/src/media/hls/parse-multivariant.ts index a7278e50..da7d97de 100644 --- a/packages/spf/src/media/hls/parse-multivariant.ts +++ b/packages/spf/src/media/hls/parse-multivariant.ts @@ -49,6 +49,7 @@ export function parseMultivariantPlaylist(text: string, unresolved: AddressableO uri?: string | undefined; default?: boolean | undefined; autoselect?: boolean | undefined; + channels?: number | undefined; } interface SubtitleRenditionInfo { @@ -101,6 +102,10 @@ export function parseMultivariantPlaylist(text: string, unresolved: AddressableO uri: uri ? resolveUrl(uri, baseUrl) : undefined, default: mediaAttrs.getBool('DEFAULT'), autoselect: mediaAttrs.getBool('AUTOSELECT'), + // CHANNELS is a quoted string whose first parameter is the channel + // count ("6", or "16/JOC" for spatial audio); getInt reads the + // leading integer. + channels: mediaAttrs.getInt('CHANNELS'), }); } @@ -175,8 +180,29 @@ export function parseMultivariantPlaylist(text: string, unresolved: AddressableO } } - // Build PartiallyResolvedVideoTracks from video streams - const videoTracks: PartiallyResolvedVideoTrack[] = videoStreams.map((stream) => { + // Build PartiallyResolvedVideoTracks from video streams, de-duplicating the + // HLS cross-product: one video rendition is listed across several + // `EXT-X-STREAM-INF` entries — one per audio group it can pair with, all + // sharing the same media-playlist URI. Collapse them to one track per URI, + // accumulating every advertised audio group. (Redundant-stream renditions + // live at *distinct* per-CDN URIs, so they stay separate — only the same-URI + // cross-product merges.) + const videoTracksByUrl = new Map(); + for (const stream of videoStreams) { + const existing = videoTracksByUrl.get(stream.uri); + if (existing) { + if (stream.audioGroupId && !existing.audioGroupIds?.includes(stream.audioGroupId)) { + existing.audioGroupIds = [...(existing.audioGroupIds ?? []), stream.audioGroupId]; + } + // BANDWIDTH is video + audio combined; the duplicates differ only in the + // paired audio. Keep the lowest as the closest proxy to video-only, which + // is what ABR should rank on. + if (stream.bandwidth < existing.bandwidth) { + existing.bandwidth = stream.bandwidth; + } + continue; + } + const codecs = stream.codecs ? parseCodecs(stream.codecs) : undefined; const track: PartiallyResolvedVideoTrack = { @@ -202,11 +228,12 @@ export function parseMultivariantPlaylist(text: string, unresolved: AddressableO track.frameRate = stream.frameRate; } if (stream.audioGroupId) { - track.audioGroupId = stream.audioGroupId; + track.audioGroupIds = [stream.audioGroupId]; } - return track; - }); + videoTracksByUrl.set(stream.uri, track); + } + const videoTracks: PartiallyResolvedVideoTrack[] = [...videoTracksByUrl.values()]; // Build PartiallyResolvedAudioTracks from audio-only streams const audioOnlyTracks: PartiallyResolvedAudioTrack[] = audioOnlyStreams.map((stream) => { @@ -252,7 +279,7 @@ export function parseMultivariantPlaylist(text: string, unresolved: AddressableO mimeType: 'audio/mp4', bandwidth: 0, // Not available in multivariant for demuxed audio sampleRate: 48000, // CMAF default - channels: 2, // Stereo default + channels: rendition.channels ?? 2, // From EXT-X-MEDIA CHANNELS; stereo default codecs: [], }; diff --git a/packages/spf/src/media/hls/tests/parse-attributes.test.ts b/packages/spf/src/media/hls/tests/parse-attributes.test.ts new file mode 100644 index 00000000..630b4555 --- /dev/null +++ b/packages/spf/src/media/hls/tests/parse-attributes.test.ts @@ -0,0 +1,33 @@ +import { describe, expect, it } from 'vitest'; +import { parseCodecs } from '../parse-attributes'; + +describe('parseCodecs', () => { + it('splits a muxed video + audio CODECS string', () => { + expect(parseCodecs('avc1.640028,mp4a.40.2')).toEqual({ video: 'avc1.640028', audio: 'mp4a.40.2' }); + }); + + it('recognizes HEVC video codecs', () => { + expect(parseCodecs('hvc1.1.6.L120.B0')).toEqual({ video: 'hvc1.1.6.L120.B0' }); + expect(parseCodecs('hev1.2.4.L120.B0')).toEqual({ video: 'hev1.2.4.L120.B0' }); + }); + + it('recognizes Dolby audio codecs (ac-3 / ec-3 / ac-4), including muxed with video', () => { + expect(parseCodecs('ac-3')).toEqual({ audio: 'ac-3' }); + expect(parseCodecs('ec-3')).toEqual({ audio: 'ec-3' }); + // The 5.1-surround Mux manifest shape: video + AC-3 audio on one STREAM-INF. + expect(parseCodecs('avc1.640020,ac-3')).toEqual({ video: 'avc1.640020', audio: 'ac-3' }); + expect(parseCodecs('avc1.640020,ac-4')).toEqual({ video: 'avc1.640020', audio: 'ac-4' }); + }); + + it('recognizes Opus / FLAC / DTS / ALAC / Vorbis audio (case-insensitive)', () => { + expect(parseCodecs('opus').audio).toBe('opus'); + expect(parseCodecs('fLaC').audio).toBe('fLaC'); + expect(parseCodecs('dtsc').audio).toBe('dtsc'); + expect(parseCodecs('alac').audio).toBe('alac'); + expect(parseCodecs('vorbis').audio).toBe('vorbis'); + }); + + it('leaves both undefined for an unrecognized codec', () => { + expect(parseCodecs('wxyz.1')).toEqual({}); + }); +}); diff --git a/packages/spf/src/media/hls/tests/parse-media-playlist.test.ts b/packages/spf/src/media/hls/tests/parse-media-playlist.test.ts index c3d5edc0..7fdadccc 100644 --- a/packages/spf/src/media/hls/tests/parse-media-playlist.test.ts +++ b/packages/spf/src/media/hls/tests/parse-media-playlist.test.ts @@ -296,4 +296,91 @@ subtitle.vtt expect(result.initialization).toBeUndefined(); }); }); + + describe('container detection (non-fMP4)', () => { + const unresolvedVideo: PartiallyResolvedVideoTrack = { + type: 'video', + id: 'video-0', + url: 'https://example.com/video/playlist.m3u8', + bandwidth: 1400000, + codecs: ['avc1.4d401f'], + mimeType: 'video/mp4', + }; + const unresolvedAudio: PartiallyResolvedAudioTrack = { + type: 'audio', + id: 'audio-0', + url: 'https://example.com/audio/playlist.m3u8', + bandwidth: 128000, + codecs: ['mp4a.40.2'], + groupId: 'audio', + name: 'Default', + sampleRate: 48000, + channels: 2, + mimeType: 'audio/mp4', + }; + + it('relabels to video/mp2t when there is no EXT-X-MAP and segments are .ts', () => { + const playlist = `#EXTM3U +#EXT-X-TARGETDURATION:6 +#EXTINF:6.0, +segment0.ts +#EXTINF:6.0, +segment1.ts +#EXT-X-ENDLIST`; + expect(parseMediaPlaylist(playlist, unresolvedVideo).mimeType).toBe('video/mp2t'); + }); + + it('uses video/mp2t for audio TS renditions too (no audio/mp2t)', () => { + const playlist = `#EXTM3U +#EXTINF:6.0, +a0.ts +#EXT-X-ENDLIST`; + expect(parseMediaPlaylist(playlist, unresolvedAudio).mimeType).toBe('video/mp2t'); + }); + + it('ignores the query string when checking the .ts extension', () => { + const playlist = `#EXTM3U +#EXTINF:6.0, +https://cdn.example.com/path/segment0.ts?token=abc123&expires=1 +#EXT-X-ENDLIST`; + expect(parseMediaPlaylist(playlist, unresolvedVideo).mimeType).toBe('video/mp2t'); + }); + + it('keeps the fMP4 default when an EXT-X-MAP init segment is present (even with a .ts-less map)', () => { + const playlist = `#EXTM3U +#EXT-X-MAP:URI="init.mp4" +#EXTINF:6.0, +segment0.ts +#EXT-X-ENDLIST`; + // EXT-X-MAP present ⇒ fMP4 by definition; never relabel. + expect(parseMediaPlaylist(playlist, unresolvedVideo).mimeType).toBe('video/mp4'); + }); + + it('relabels to audio/aac when there is no EXT-X-MAP and segments are .aac (raw ADTS)', () => { + const playlist = `#EXTM3U +#EXTINF:9.98, +fileSequence0.aac +#EXTINF:9.98, +fileSequence1.aac +#EXT-X-ENDLIST`; + expect(parseMediaPlaylist(playlist, unresolvedAudio).mimeType).toBe('audio/aac'); + }); + + it('keeps the fMP4 default for an .aac rendition that has an EXT-X-MAP', () => { + const playlist = `#EXTM3U +#EXT-X-MAP:URI="init.mp4" +#EXTINF:9.98, +fileSequence0.aac +#EXT-X-ENDLIST`; + expect(parseMediaPlaylist(playlist, unresolvedAudio).mimeType).toBe('audio/mp4'); + }); + + it('keeps the fMP4 default when there is no map but the extension is unrecognized (e.g. .mp4)', () => { + const playlist = `#EXTM3U +#EXTINF:6.0, +segment0.mp4 +#EXT-X-ENDLIST`; + expect(parseMediaPlaylist(playlist, unresolvedVideo).mimeType).toBe('video/mp4'); + }); + }); }); diff --git a/packages/spf/src/media/hls/tests/parse-multivariant.test.ts b/packages/spf/src/media/hls/tests/parse-multivariant.test.ts index 070d3478..1fb8529d 100644 --- a/packages/spf/src/media/hls/tests/parse-multivariant.test.ts +++ b/packages/spf/src/media/hls/tests/parse-multivariant.test.ts @@ -74,7 +74,43 @@ video-1080p.m3u8`; expect(typeof track1080p.id).toBe('string'); // Optional fields not present expect(track1080p.frameRate).toBeUndefined(); - expect(track1080p.audioGroupId).toBeUndefined(); + expect(track1080p.audioGroupIds).toBeUndefined(); + }); + + it('de-duplicates the EXT-X-STREAM-INF cross-product: one track per video URI, accumulating audio groups', () => { + // The same two video renditions are each listed twice — once paired with a + // 5.1 group (higher BANDWIDTH, ac-3) and once with a stereo group (lower, + // mp4a) — sharing the same media-playlist URI. Mirrors the Mux + // enable_51_surround shape. + const text = `#EXTM3U +#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="audio-51",NAME="Default",CHANNELS="6",URI="https://example.com/a51.m3u8" +#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="audio-hi",NAME="Default",CHANNELS="2",URI="https://example.com/ahi.m3u8" +#EXT-X-STREAM-INF:BANDWIDTH=3200000,RESOLUTION=1920x1080,CODECS="avc1.640028,ac-3",AUDIO="audio-51" +https://example.com/v1080.m3u8 +#EXT-X-STREAM-INF:BANDWIDTH=2800000,RESOLUTION=1920x1080,CODECS="mp4a.40.2,avc1.640028",AUDIO="audio-hi" +https://example.com/v1080.m3u8 +#EXT-X-STREAM-INF:BANDWIDTH=1600000,RESOLUTION=1280x720,CODECS="avc1.4d401f,ac-3",AUDIO="audio-51" +https://example.com/v720.m3u8 +#EXT-X-STREAM-INF:BANDWIDTH=1400000,RESOLUTION=1280x720,CODECS="mp4a.40.2,avc1.4d401f",AUDIO="audio-hi" +https://example.com/v720.m3u8`; + + const result = parseMultivariantPlaylist(text, { url: baseUrl }); + const videoTracks = (result.selectionSets.find((s) => s.type === 'video')?.switchingSets[0]?.tracks ?? + []) as PartiallyResolvedVideoTrack[]; + + // Two unique URIs → two tracks, not four. + expect(videoTracks).toHaveLength(2); + + const v1080 = videoTracks.find((t) => t.url === 'https://example.com/v1080.m3u8'); + expect(v1080?.audioGroupIds).toEqual(['audio-51', 'audio-hi']); + // Lowest combined BANDWIDTH across the duplicates (the stereo pairing). + expect(v1080?.bandwidth).toBe(2800000); + // Video codec is identical across the duplicates regardless of paired audio. + expect(v1080?.codecs).toEqual(['avc1.640028']); + + const v720 = videoTracks.find((t) => t.url === 'https://example.com/v720.m3u8'); + expect(v720?.audioGroupIds).toEqual(['audio-51', 'audio-hi']); + expect(v720?.bandwidth).toBe(1400000); }); it('handles relative URLs by resolving against baseUrl', () => { @@ -225,7 +261,7 @@ video-lo.m3u8`; width: 768, height: 432, codecs: ['avc1.64001f'], - audioGroupId: 'audio-med-0', + audioGroupIds: ['audio-med-0'], }); expect(videoTracks?.[1]).toMatchObject({ @@ -234,7 +270,7 @@ video-lo.m3u8`; width: 640, height: 360, codecs: ['avc1.64001f'], - audioGroupId: 'audio-lo-0', + audioGroupIds: ['audio-lo-0'], }); }); @@ -252,6 +288,25 @@ video-lo.m3u8`; ]); }); + it('parses the CHANNELS attribute on audio renditions (6 for 5.1, default 2 when absent)', () => { + const text = `#EXTM3U +#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="surround",NAME="5.1",CHANNELS="6",URI="https://example.com/a51.m3u8" +#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="stereo",NAME="Stereo",CHANNELS="2",URI="https://example.com/a2.m3u8" +#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="nochan",NAME="Unspecified",URI="https://example.com/anc.m3u8" +#EXT-X-STREAM-INF:BANDWIDTH=3000000,RESOLUTION=1920x1080,CODECS="avc1.640028,ac-3",AUDIO="surround" +https://example.com/v.m3u8`; + + const result = parseMultivariantPlaylist(text, { url: baseUrl }); + const audioTracks = (result.selectionSets.find((s) => s.type === 'audio')?.switchingSets[0]?.tracks ?? + []) as PartiallyResolvedAudioTrack[]; + const byGroup = (groupId: string) => audioTracks.find((t) => t.groupId === groupId); + + expect(byGroup('surround')?.channels).toBe(6); + expect(byGroup('stereo')?.channels).toBe(2); + // No CHANNELS attribute → stereo default. + expect(byGroup('nochan')?.channels).toBe(2); + }); + it('extracts audio codecs from referencing streams', () => { const result = parseMultivariantPlaylist(muxPlaylist, { url: baseUrl }); const audioSet = result.selectionSets.find((s) => s.type === 'audio'); @@ -517,9 +572,11 @@ v2/prog_index.m3u8 expect(videoSet).toBeDefined(); expect(videoSet!.switchingSets).toBeDefined(); - // Should have 24 video variants (8 resolutions × 3 codec combinations) + // 8 unique video renditions. The manifest lists them as 24 EXT-X-STREAM-INF + // entries (8 URIs × 3 audio groups: mp4a / ac-3 / ec-3), the HLS + // cross-product; the parser de-duplicates by URI to one track per rendition. const allVideoTracks = videoSet!.switchingSets.flatMap((ss) => ss.tracks); - expect(allVideoTracks.length).toBe(24); + expect(allVideoTracks.length).toBe(8); // Check first track details (960x540, 60fps, avc1+mp4a, aud1) const firstTrack = allVideoTracks.find((t) => t.bandwidth === 2177116); diff --git a/packages/spf/src/media/types/index.ts b/packages/spf/src/media/types/index.ts index 4a3c4a38..7aecbbf2 100644 --- a/packages/spf/src/media/types/index.ts +++ b/packages/spf/src/media/types/index.ts @@ -144,7 +144,14 @@ export type VideoTrack = Track & width?: number; height?: number; frameRate?: FrameRate; - audioGroupId?: string; + /** + * Audio groups (`EXT-X-STREAM-INF:AUDIO`) this video rendition can pair + * with. A list because one rendition is typically listed across multiple + * `EXT-X-STREAM-INF` entries — one per audio group (the HLS cross-product) — + * which the parser collapses into a single track carrying every group it + * advertised. + */ + audioGroupIds?: string[]; }; /** @@ -174,6 +181,21 @@ export type TextTrack = Track & { forced?: boolean; }; +/** + * Predicate that answers "can this environment decode this track?" — the + * capability-probing surface, read by the track-switching hard-constraint + * pre-pass (`excludeUnplayableTracks`) to drop undecodable renditions before + * selection. Kept DOM-free here (a plain function type over a minimal track + * shape) so DOM-free behaviors can consume it; the DOM implementation + * (`canPlayTrack` in `media/dom/capabilities.ts`) wraps + * `MediaSource.isTypeSupported`. + * + * Takes the minimal codec-bearing shape both video and audio candidates + * carry. `mimeType` is optional so unprobeable candidates (no MIME) can be + * passed straight through as playable rather than dropped. + */ +export type CanPlayTrack = (track: { mimeType?: string; codecs?: string[] }) => boolean; + /** * Minimal text-track cue shape — start time, end time, and display text. * diff --git a/packages/spf/src/media/utils/tests/tracks.test.ts b/packages/spf/src/media/utils/tests/tracks.test.ts new file mode 100644 index 00000000..7fb2e9cd --- /dev/null +++ b/packages/spf/src/media/utils/tests/tracks.test.ts @@ -0,0 +1,51 @@ +import { describe, expect, it } from 'vitest'; +import type { Presentation } from '../../types'; +import { applyContainerMimeType } from '../tracks'; + +const presentation = (): Presentation => + ({ + id: 'pres-1', + url: 'https://example.com/master.m3u8', + selectionSets: [ + { + id: 'v', + type: 'video', + switchingSets: [ + { + id: 'vs', + type: 'video', + tracks: [ + { id: 'v1', mimeType: 'video/mp4' }, + { id: 'v2', mimeType: 'video/mp4' }, + ], + }, + ], + }, + { + id: 'a', + type: 'audio', + switchingSets: [{ id: 'as', type: 'audio', tracks: [{ id: 'a1', mimeType: 'audio/mp4' }] }], + }, + ], + }) as unknown as Presentation; + +const mimeOf = (p: Presentation, type: string) => + p.selectionSets.find((s) => s.type === type)?.switchingSets[0]?.tracks.map((t) => t.mimeType); + +describe('applyContainerMimeType', () => { + it('sets the MIME on every track of the given type', () => { + const result = applyContainerMimeType(presentation(), 'video', 'video/mp2t'); + expect(mimeOf(result, 'video')).toEqual(['video/mp2t', 'video/mp2t']); + }); + + it('leaves other types untouched (never crosses audio↔video)', () => { + const result = applyContainerMimeType(presentation(), 'video', 'video/mp2t'); + expect(mimeOf(result, 'audio')).toEqual(['audio/mp4']); + }); + + it('is idempotent — re-applying the same MIME is a no-op', () => { + const once = applyContainerMimeType(presentation(), 'video', 'video/mp2t'); + const twice = applyContainerMimeType(once, 'video', 'video/mp2t'); + expect(twice).toEqual(once); + }); +}); diff --git a/packages/spf/src/media/utils/tracks.ts b/packages/spf/src/media/utils/tracks.ts index 5db8ca37..379e876e 100644 --- a/packages/spf/src/media/utils/tracks.ts +++ b/packages/spf/src/media/utils/tracks.ts @@ -119,6 +119,37 @@ export function hasCodecs(track: PartiallyResolvedTrack | ResolvedTrack | undefi return !!track && 'codecs' in track && !!track.codecs?.length; } +/** + * Set `mimeType` on every track of one `type` (immutably). Used to propagate a + * detected container across a type's renditions: an ABR ladder is the same + * content at different bitrates, so one rendition's container holds for all of + * them — capability probing + SourceBuffer setup then get the right MIME for the + * whole type from a single resolved media playlist, without fetching the rest. + * + * Scoped to one type on purpose: propagating *across* audio/video would be wrong + * for mixed-container sources (e.g. muxed-TS video + raw-`.aac` audio) and races + * concurrent per-type resolution. Same-type writes are disjoint and safe. + * Idempotent — tracks already at `mimeType` are left as-is. + */ +export function applyContainerMimeType(presentation: Presentation, type: TrackType, mimeType: string): Presentation { + return { + ...presentation, + selectionSets: presentation.selectionSets.map((selectionSet) => + selectionSet.type === type + ? { + ...selectionSet, + switchingSets: selectionSet.switchingSets.map((switchingSet) => ({ + ...switchingSet, + tracks: switchingSet.tracks.map((track) => + track.mimeType === mimeType ? track : { ...track, mimeType } + ), + })), + } + : selectionSet + ), + } as Presentation; +} + /** * Updates a track within a presentation (immutably). Generic — works for * video, audio, or text tracks. diff --git a/packages/spf/src/playback/behaviors/resolve-track.ts b/packages/spf/src/playback/behaviors/resolve-track.ts index a49712b8..fbcb68ff 100644 --- a/packages/spf/src/playback/behaviors/resolve-track.ts +++ b/packages/spf/src/playback/behaviors/resolve-track.ts @@ -2,11 +2,11 @@ import { defineBehavior } from '../../core/composition/create-composition'; import { createMachineReactor } from '../../core/reactors/create-machine-reactor'; import { computed, peek, type ReadonlySignal, type Signal, update } from '../../core/signals/primitives'; import { ConcurrentRunner, Task } from '../../core/tasks/task'; -import { parseMediaPlaylist } from '../../media/hls/parse-media-playlist'; +import { NON_FMP4_CONTAINER_MIMES, parseMediaPlaylist } from '../../media/hls/parse-media-playlist'; import type { MaybeResolvedPresentation, PartiallyResolvedTrack, ResolvedTrack } from '../../media/types'; import { isResolvedPresentation, isResolvedTrack } from '../../media/types'; import type { GetCdnId } from '../../media/utils/cdn'; -import { findTrack, updateTrackInPresentation } from '../../media/utils/tracks'; +import { applyContainerMimeType, findTrack, updateTrackInPresentation } from '../../media/utils/tracks'; import { fetchResolvableText as defaultFetchResolvableText, type FetchText } from '../../network/fetch'; import { failoverFetch } from '../primitives/failover-fetch'; import { AUDIO_TYPE_CONFIG, TEXT_TYPE_CONFIG, VIDEO_TYPE_CONFIG } from '../primitives/track-types'; @@ -132,9 +132,21 @@ function setupTrackResolution({ // signal abort cancels in-flight body reads — so by the // time we reach this point the presentation we resolved // against is the live one. - update(state.presentation, (current) => - isResolvedPresentation(current) ? updateTrackInPresentation(current, mediaTrack) : current - ); + update(state.presentation, (current) => { + if (!isResolvedPresentation(current)) return current; + const patched = updateTrackInPresentation(current, mediaTrack); + // Container is uniform within a type (an ABR ladder shares + // its container), so a detected non-fMP4 rendition (TS, + // raw AAC) implies every rendition of *this* type matches — + // relabel them all from one resolved playlist instead of + // fetching each. Scoped to this track's own type: never cross + // audio↔video (mixed-container sources exist, e.g. muxed-TS + // video + raw-.aac audio), which also keeps per-type + // resolutions' writes disjoint (no race). + return NON_FMP4_CONTAINER_MIMES.has(mediaTrack.mimeType) + ? applyContainerMimeType(patched, mediaTrack.type, mediaTrack.mimeType) + : patched; + }); }, { id: track.id } ) diff --git a/packages/spf/src/playback/behaviors/tests/track-switching.test.ts b/packages/spf/src/playback/behaviors/tests/track-switching.test.ts index 0c88a5ff..d245a278 100644 --- a/packages/spf/src/playback/behaviors/tests/track-switching.test.ts +++ b/packages/spf/src/playback/behaviors/tests/track-switching.test.ts @@ -10,6 +10,7 @@ import type { VideoSelectionSet, VideoTrack, } from '../../../media/types'; +import { applyContainerMimeType } from '../../../media/utils/tracks'; import type { BandwidthState } from '../../../network/bandwidth-estimator'; import { applyConstraints, @@ -969,19 +970,126 @@ describe('excludeFailedCdns (failover constraint)', () => { reactor.destroy(); }); - it('keeps the prior pick when every CDN is in cooldown (nothing playable)', async () => { + it('clears the selection when every CDN is in cooldown, re-picking on recovery', async () => { + const errorSpy = vi.spyOn(console, 'error').mockImplementation(() => {}); const state = makeState(undefined); const reactor = switchVideoTrack.setup({ state }); await flush(); expect(state.selectedVideoTrackId.get()).toBe('1080p-a'); - // All CDNs cooled down → constraints prune everything → no playable set → - // the effect no-ops, leaving the last pick in place (deferred terminal-state - // modeling). + // All CDNs cooled down → constraints prune every track → no playable set → + // the selection clears (no pick) rather than lingering on an unreachable CDN. state.failedCdns.set(['https://cdn-a.example.com', 'https://cdn-b.example.com']); await flush(); + expect(state.selectedVideoTrackId.get()).toBeUndefined(); + expect(errorSpy).toHaveBeenCalled(); + + // A CDN recovers → its tracks reappear → the candidate set refills and re-picks. + state.failedCdns.set(['https://cdn-b.example.com']); + await flush(); expect(state.selectedVideoTrackId.get()).toBe('1080p-a'); + errorSpy.mockRestore(); + reactor.destroy(); + }); +}); + +// ============================================================================ +// excludeUnplayableTracks — the capability constraint (hard pre-pass) +// ============================================================================ + +describe('excludeUnplayableTracks (capability constraint)', () => { + const codecVideoTrack = (id: string, codec: string, bandwidth: number): PartiallyResolvedVideoTrack => ({ + type: 'video', + codecs: [codec], + id, + url: `https://example.com/${id}.m3u8`, + bandwidth, + mimeType: 'video/mp4', + }); + + // Mixed HEVC + AVC ladder; the same bitrates on each codec. + const mixedCodecPresentation = () => + createPresentation([ + codecVideoTrack('720p-hevc', 'hvc1.1.6.L93.B0', 2_400_000), + codecVideoTrack('720p-avc', 'avc1.4d401f', 2_400_000), + codecVideoTrack('1080p-hevc', 'hvc1.1.6.L120.B0', 4_800_000), + codecVideoTrack('1080p-avc', 'avc1.640028', 4_800_000), + ]); + + // Rejects HEVC, accepts everything else. + const rejectsHevc = (track: { codecs?: string[] }) => !track.codecs?.some((c) => c.startsWith('hvc1')); + + const makeState = () => ({ + presentation: signal(mixedCodecPresentation()), + bandwidthState: signal(createBandwidthState(10_000_000)), + selectedVideoTrackId: signal(undefined), + userVideoTrackSelection: signal | undefined>(undefined), + }); + + it('prunes undecodable renditions before ranking — picks the best playable codec', async () => { + const state = makeState(); + const reactor = switchVideoTrack.setup({ state, config: { canPlayTrack: rejectsHevc } }); + await flush(); + // HEVC pruned upstream; ranker picks the highest-bitrate AVC that fits. + expect(state.selectedVideoTrackId.get()).toBe('1080p-avc'); + reactor.destroy(); + }); + + it('passes everything through when no canPlayTrack probe is wired', async () => { + const state = makeState(); + const reactor = switchVideoTrack.setup({ state }); + await flush(); + // No probe → HEVC survives; same-bitrate tie keeps manifest order, so the + // first 1080p (HEVC) wins. + expect(state.selectedVideoTrackId.get()).toBe('1080p-hevc'); + reactor.destroy(); + }); + + it('still excludes an unplayable track the user selected (hard constraint beats the soft filter)', async () => { + const state = makeState(); + state.userVideoTrackSelection.set({ id: '1080p-hevc' }); + const reactor = switchVideoTrack.setup({ state, config: { canPlayTrack: rejectsHevc } }); + await flush(); + // The user's HEVC pick is pruned by the constraint before the user filter + // runs; the filter finds no match and falls through to the playable set. + expect(state.selectedVideoTrackId.get()).toBe('1080p-avc'); + reactor.destroy(); + }); + + it('makes no pick when the constraint prunes every rendition', async () => { + const errorSpy = vi.spyOn(console, 'error').mockImplementation(() => {}); + const state = makeState(); + const reactor = switchVideoTrack.setup({ state, config: { canPlayTrack: () => false } }); + await flush(); + // Every codec rejected from a cold start → empty candidate set → nothing + // selected, and the empty-from-constraints case is flagged. The late + // createSourceBuffer check stays as the backstop. + expect(state.selectedVideoTrackId.get()).toBeUndefined(); + expect(errorSpy).toHaveBeenCalled(); + errorSpy.mockRestore(); + reactor.destroy(); + }); + + it('clears a prior pick when a later relabel prunes every rendition to empty', async () => { + const errorSpy = vi.spyOn(console, 'error').mockImplementation(() => {}); + const state = makeState(); + // Accepts fMP4; rejects the non-fMP4 container MIME resolve-track relabels to. + const canPlayTrack = (track: { mimeType?: string }) => track.mimeType !== 'video/mp2t'; + const reactor = switchVideoTrack.setup({ state, config: { canPlayTrack } }); + await flush(); + // Pick made while the tracks are still labeled video/mp4. + expect(state.selectedVideoTrackId.get()).toBe('1080p-hevc'); + + // resolve-track detects a TS container and relabels the whole video type; + // every rendition is now undecodable → candidate set empties → the now-stale + // pick clears (instead of lingering as an unplayable selection that stalls). + state.presentation.set(applyContainerMimeType(mixedCodecPresentation(), 'video', 'video/mp2t')); + await flush(); + expect(state.selectedVideoTrackId.get()).toBeUndefined(); + expect(errorSpy).toHaveBeenCalled(); + + errorSpy.mockRestore(); reactor.destroy(); }); }); diff --git a/packages/spf/src/playback/behaviors/track-switching.ts b/packages/spf/src/playback/behaviors/track-switching.ts index 6ed8cf82..d989851c 100644 --- a/packages/spf/src/playback/behaviors/track-switching.ts +++ b/packages/spf/src/playback/behaviors/track-switching.ts @@ -5,10 +5,10 @@ * unload. * * Selection runs in two stages. First a **hard-constraints pre-pass** - * (`applyConstraints`) prunes the unplayable from the candidate set — today the - * failed-CDN constraint (`excludeFailedCdns`, failover cooldown); capability - * probing will join it. Then a small ordered chain of rules (`applyRules`) picks - * among the survivors. Each constraint/rule reads the signals it needs at apply + * (`applyConstraints`) prunes the unplayable from the candidate set — the + * failed-CDN constraint (`excludeFailedCdns`, failover cooldown) and the + * capability constraint (`excludeUnplayableTracks`, codec support). Then a small + * ordered chain of rules (`applyRules`) picks among the survivors. Each constraint/rule reads the signals it needs at apply * time, so the effect subscribes to exactly what was consulted. The chain is * three rules, most authoritative first: * @@ -41,18 +41,25 @@ * *scope* is the sticky-pick half of multi-CDN; the failed-CDN *constraint* is * the failover half — prune the cooled-down CDN, the scope falls to the next.) * - * Deferred: capability probing as a second constraint; audio's preferred- - * language / default-track selection as standing soft-filter rules (previously - * the empty-slot picker, dropped in the move to the rule chain). + * When the pre-pass prunes a type's candidates to empty, the behavior leaves any + * prior pick in place and makes no new pick; the late `createSourceBuffer` check + * stays as the structural backstop for an unplayable rendition reaching the + * pipeline. Surfacing "nothing playable" as observable state is deferred until a + * consumer (error mapping) needs it. + * + * Deferred: audio's preferred-language / default-track selection as standing + * soft-filter rules (previously the empty-slot picker, dropped in the move to + * the rule chain). */ import { type AnySlotMap, defineBehavior } from '../../core/composition/create-composition'; import { createMachineReactor } from '../../core/reactors/create-machine-reactor'; -import { computed, type ReadonlySignal, type Signal } from '../../core/signals/primitives'; +import { computed, peek, type ReadonlySignal, type Signal } from '../../core/signals/primitives'; import { DEFAULT_QUALITY_CONFIG, type QualityConfig, resolutionArea } from '../../media/abr/quality-selection'; import { matchesPartialTrack } from '../../media/primitives/select-tracks'; import { type AudioTrack, + type CanPlayTrack, isResolvedPresentation, type MaybeResolvedPresentation, type PartiallyResolvedAudioTrack, @@ -96,6 +103,15 @@ export interface SwitchVideoTrackConfig { initialBandwidth?: number; /** Override CDN-id derivation (shared by the CDN scope + failover constraint). */ getCdnId?: GetCdnId; + /** + * Codec capability probe read by the `excludeUnplayableTracks` hard + * constraint — drops renditions this environment can't decode before + * selection runs. Injected (rather than imported) so the DOM-free behavior + * never reaches a DOM API directly; the engine defaults it to the + * `MediaSource.isTypeSupported`-backed `canPlayTrack`. Absent → no codec + * filtering (the constraint passes everything through). + */ + canPlayTrack?: CanPlayTrack; } /** Default initial-bandwidth value before bandwidth measurements arrive. */ @@ -222,7 +238,18 @@ export function applyConstraints( * audio candidates area-compare equal). Every resolved/partially-resolved video * track carries them all; audio tracks omit the dimensions. */ -type SwitchableTrack = { id: string; url: string; bandwidth?: number; width?: number; height?: number }; +type SwitchableTrack = { + id: string; + url: string; + bandwidth?: number; + width?: number; + height?: number; + // Read by the capability constraint to probe codec support. Optional on the + // minimal shape; every resolved/partially-resolved video & audio candidate + // carries them, and an absent `mimeType` makes a track unprobeable (kept). + mimeType?: string; + codecs?: string[]; +}; type SelectionKey = 'selectedVideoTrackId' | 'selectedAudioTrackId'; type UserSelectionKey = 'userVideoTrackSelection' | 'userAudioTrackSelection'; @@ -339,6 +366,17 @@ type CdnRuleConfig = TrackSwi getCdnId?: GetCdnId; }; +/** + * Config the capability constraint reads: the base config plus an *optional* + * `canPlayTrack` codec probe. Optional → an unwired probe means "no codec + * filtering" and the constraint passes everything through, so the base config + * (without it) stays assignable. The engine defaults it to the DOM-bound + * `canPlayTrack`. + */ +type CapabilityConstraintConfig = TrackSwitchingConfig & { + canPlayTrack?: CanPlayTrack; +}; + type VideoTrackCandidate = PartiallyResolvedVideoTrack | VideoTrack; type AudioTrackCandidate = PartiallyResolvedAudioTrack | AudioTrack; @@ -373,8 +411,8 @@ function filterByUserSelection( tracks: readonly T[], @@ -387,6 +425,31 @@ function excludeFailedCdns( return tracks.filter((track) => !failedSet.has(getCdnId(track.url))); } +/** + * Capability constraint — a *hard* filter (constraints pre-pass), shared by + * video and audio. Removes renditions this environment can't decode, probed via + * the injected `canPlayTrack` (codec → `MediaSource.isTypeSupported`). Moving + * the check here — before selection — means an unplayable variant (e.g. HEVC on + * a browser without HEVC) is pruned upstream and never picked, instead of + * surviving into the pipeline to fail late at `createSourceBuffer`. That late + * throw stays as a defensive structural guarantee; with this constraint it + * should rarely fire. + * + * Passes everything through when there's no `canPlayTrack` probe (a composition + * that didn't wire it, or DOM-free tests). When it prunes *every* track (no + * decodable rendition), the empty result is preserved (per `applyConstraints`) + * — "nothing playable," so the behavior clears the selection (no pick) and the + * late `createSourceBuffer` check stays as the backstop. + */ +function excludeUnplayableTracks( + tracks: readonly T[], + { config }: SelectionRuleDeps, AnySlotMap, CapabilityConstraintConfig> +): readonly T[] { + const canPlay = config.canPlayTrack; + if (!canPlay) return tracks; + return tracks.filter((track) => canPlay(track)); +} + /** * Active-CDN scope — a soft filter, shared by video and audio. Narrows to the * highest-priority CDN in `cdnPriority` (owned by `deriveCdnPriority`) that @@ -532,14 +595,34 @@ function setupTrackSwitching< effects: [ () => { // Reactive read: subscribes the reaction to the candidate set, so a - // new presentation — or a future constraint pruning it — re-fires - // this and re-picks. + // new presentation — or a constraint pruning it — re-fires this and + // re-picks. const tracks = candidateSet.get(); - // No playable tracks: no tracks of this type, or (once constraints - // exist) everything pruned. Nothing to pick. Surfacing "nothing - // playable" as a distinct not-ready state is left to the constraints - // work; for now it's a silent no-op, leaving any prior pick in place. - if (!tracks.length) return; + + // Empty candidate set — two shapes, told apart by whether the type + // has any tracks at all: + // - The type has no tracks (e.g. a video-only source's absent + // audio): legitimate, nothing to pick or clear. + // - The type HAS tracks but the hard-constraints pre-pass pruned + // every one (every rendition undecodable, or every CDN in + // failover cooldown): no playable rendition. Clear the selection + // so a pick made earlier — e.g. under the initial mp4 label, + // before resolve-track relabeled the type to a non-fMP4 + // container — can't linger as a now-unplayable selection and + // silently stall the pipeline. + // The `console.error` is a placeholder until the planned error + // behaviors surface "nothing playable" as observable state. + if (!tracks.length) { + const presentation = peek(state.presentation); + const hasTracksOfType = isResolvedPresentation(presentation) && getTracks(presentation).length > 0; + if (hasTracksOfType) { + console.error( + `[track-switching] every ${selectionKey} candidate was filtered out by constraints; clearing selection` + ); + state[selectionKey].set(undefined); + } + return; + } // The whole deps object passes straight through to every rule in the // variant-supplied chain (state + config from the behavior; context @@ -602,7 +685,7 @@ export const switchVideoTrack = defineBehavior({ selectionKey: 'selectedVideoTrackId', userSelectionKey: 'userVideoTrackSelection', getTracks: (presentation) => getTracksByType(presentation, 'video') as readonly VideoTrackCandidate[], - constraints: [excludeFailedCdns], + constraints: [excludeFailedCdns, excludeUnplayableTracks], rules: [filterByUserSelection, preferActiveCdn, rankByBandwidth], }, }), @@ -653,7 +736,7 @@ export const switchAudioTrack = defineBehavior({ selectionKey: 'selectedAudioTrackId', userSelectionKey: 'userAudioTrackSelection', getTracks: (presentation) => getTracksByType(presentation, 'audio') as readonly AudioTrackCandidate[], - constraints: [excludeFailedCdns], + constraints: [excludeFailedCdns, excludeUnplayableTracks], rules: [filterByUserSelection, preferActiveCdn, rankByBandwidth], }, }), diff --git a/packages/spf/src/playback/engines/hls/engine-audio-only.ts b/packages/spf/src/playback/engines/hls/engine-audio-only.ts index 77fb0a4f..5bd38953 100644 --- a/packages/spf/src/playback/engines/hls/engine-audio-only.ts +++ b/packages/spf/src/playback/engines/hls/engine-audio-only.ts @@ -7,8 +7,9 @@ import { import { makeShareSignals, type ShareSignalsConfig } from '../../../core/composition/share-signals'; import type { BackBufferConfig } from '../../../media/buffer/back-buffer'; import type { ForwardBufferConfig } from '../../../media/buffer/forward-buffer'; +import { canPlayTrack } from '../../../media/dom/capabilities'; import { parseMultivariantPlaylist } from '../../../media/hls/parse-multivariant'; -import type { AudioTrack, MaybeResolvedPresentation } from '../../../media/types'; +import type { AudioTrack, CanPlayTrack, MaybeResolvedPresentation } from '../../../media/types'; import type { GetCdnId } from '../../../media/utils/cdn'; import { getResolvedSelectedTrackDuration } from '../../../media/utils/track-selection'; import type { SegmentLoaderActor } from '../../actors/dom/segment-loader'; @@ -98,6 +99,14 @@ export type SimpleHlsAudioOnlyEngineSignals = { export interface SimpleHlsAudioOnlyEngineConfig extends ShareSignalsConfig { preferredAudioLanguage?: string; + /** + * Codec capability probe read by `track-switching`'s `excludeUnplayableTracks` + * constraint. Defaults to the `MediaSource.isTypeSupported`-backed + * `canPlayTrack`; override to force-exclude a codec. Mirrors the default + * engine — without it, capability probing (and TS / raw-AAC detection) would + * be inert for audio-only playback. + */ + canPlayTrack?: CanPlayTrack; resolveDuration?: PresentationDurationResolver; parsePresentation?: ParsePresentation; forwardBuffer?: Partial; @@ -159,6 +168,7 @@ export function createHlsAudioOnlyEngine( ): Composition { const finalConfig = { ...config, + canPlayTrack: config.canPlayTrack ?? canPlayTrack, resolveDuration: config.resolveDuration ?? getResolvedSelectedTrackDuration, parsePresentation: config.parsePresentation ?? parseMultivariantPlaylist, }; diff --git a/packages/spf/src/playback/engines/hls/engine.ts b/packages/spf/src/playback/engines/hls/engine.ts index 874622a2..e6f992cb 100644 --- a/packages/spf/src/playback/engines/hls/engine.ts +++ b/packages/spf/src/playback/engines/hls/engine.ts @@ -8,6 +8,7 @@ import { makeShareSignals, type ShareSignalsConfig } from '../../../core/composi import type { QualityConfig } from '../../../media/abr/quality-selection'; import type { BackBufferConfig } from '../../../media/buffer/back-buffer'; import type { ForwardBufferConfig } from '../../../media/buffer/forward-buffer'; +import { canPlayTrack } from '../../../media/dom/capabilities'; import { resolveVttSegment } from '../../../media/dom/text/resolve-vtt-segment'; import { addSubtitlesTracksToMedia, @@ -15,7 +16,7 @@ import { removeAllSubtitlesTracksFromMedia, } from '../../../media/dom/text/text-track-slots'; import { parseMultivariantPlaylist } from '../../../media/hls/parse-multivariant'; -import type { AudioTrack, MaybeResolvedPresentation, VideoTrack } from '../../../media/types'; +import type { AudioTrack, CanPlayTrack, MaybeResolvedPresentation, VideoTrack } from '../../../media/types'; import type { GetCdnId } from '../../../media/utils/cdn'; import { getResolvedSelectedTrackDuration } from '../../../media/utils/track-selection'; import type { BandwidthConfig, BandwidthState } from '../../../network/bandwidth-estimator'; @@ -135,6 +136,14 @@ export interface SimpleHlsEngineConfig extends ShareSignalsConfig { const finalConfig = { ...config, + canPlayTrack: config.canPlayTrack ?? canPlayTrack, resolveTextTrackSegment: config.resolveTextTrackSegment ?? resolveVttSegment, resolveDuration: config.resolveDuration ?? getResolvedSelectedTrackDuration, parsePresentation: config.parsePresentation ?? parseMultivariantPlaylist, diff --git a/packages/spf/src/playback/engines/hls/tests/engine-audio-only.test.ts b/packages/spf/src/playback/engines/hls/tests/engine-audio-only.test.ts index b2433981..02968c9d 100644 --- a/packages/spf/src/playback/engines/hls/tests/engine-audio-only.test.ts +++ b/packages/spf/src/playback/engines/hls/tests/engine-audio-only.test.ts @@ -1,5 +1,6 @@ import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; import { snapshot } from '../../../../core/signals/primitives'; +import type { Presentation } from '../../../../media/types'; import { createHlsAudioOnlyEngine } from '../engine-audio-only'; // Mock appendSegment to succeed without real MP4 data @@ -7,15 +8,27 @@ vi.mock('../../../../media/dom/mse/append-segment', () => ({ appendSegment: vi.fn().mockResolvedValue(undefined), })); +// Fallback for URLs a test's mock doesn't handle explicitly. Segment/init +// requests resolve with an empty body — the appendSegment mock makes the bytes +// inert — so the failover monitor isn't tripped by unmocked segment fetches (a +// single failed fetch trips that CDN into cooldown, which empties the candidate +// set). Genuinely unknown URLs still reject loudly. +function unmockedFetchFallback(url: string): Promise { + // Non-empty body: `fetchStream` throws "Response has no body" on a null body + // (empty Uint8Array), which would itself trip the monitor. + if (/\.(m4s|mp4|ts|aac)(\?|$)/.test(url)) return Promise.resolve(new Response(new Uint8Array([0]))); + return Promise.reject(new Error(`Unmocked URL: ${url}`)); +} + describe('createHlsAudioOnlyEngine', () => { let originalFetch: typeof globalThis.fetch; let consoleErrorSpy: ReturnType; - // Tests assert at actor-presence and state-shape level, not at "init - // segment appended" level — so unmocked init/segment URLs in the manifests - // are intentional. The fetch loop's reject path leaks a console.error in - // each test; suppress only the expected patterns so genuine failures still - // surface. + // Tests assert at actor-presence and state-shape level, not at "init segment + // appended" level. Audio/video segment fetches resolve via + // `unmockedFetchFallback` (inert under the appendSegment mock); text-track + // segment fetches still reject and leak a console.error. Suppress only the + // expected patterns so genuine failures still surface. const expectedErrorPatterns = [ /Unexpected error in segment loader.*Unmocked URL/s, /Failed to load text-track segment/, @@ -62,6 +75,51 @@ describe('createHlsAudioOnlyEngine', () => { engine.destroy(); }); + it('wires the default canPlayTrack — prunes an undecodable (raw-AAC) audio source, making no pick', async () => { + const flush = () => Promise.resolve().then(() => Promise.resolve()); + // No canPlayTrack override → relies on the engine's default. A raw-AAC + // (audio/aac) rendition is asserted unplayable, so it should be pruned + // rather than selected. (If the default weren't wired, the constraint would + // pass through and select it.) + const engine = createHlsAudioOnlyEngine(); + engine.state.presentation.set({ + id: 'pres-aac', + url: 'https://example.com/master.m3u8', + startTime: 0, + selectionSets: [ + { + id: 'a', + type: 'audio', + switchingSets: [ + { + id: 'as', + type: 'audio', + tracks: [ + { + type: 'audio', + id: 'aud-aac', + codecs: ['mp4a.40.2'], + url: 'https://example.com/aud.m3u8', + bandwidth: 128_000, + mimeType: 'audio/aac', + groupId: 'audio', + name: 'Default', + sampleRate: 48_000, + channels: 2, + }, + ], + }, + ], + }, + ], + } as Presentation); + await flush(); + + expect(engine.state.selectedAudioTrackId.get()).toBeUndefined(); + + engine.destroy(); + }); + it('does not seed bandwidthState (no ABR behavior subscribed at init)', () => { const engine = createHlsAudioOnlyEngine(); @@ -101,7 +159,7 @@ http://example.com/audio-seg1.m4s ); } - return Promise.reject(new Error(`Unmocked URL: ${url}`)); + return unmockedFetchFallback(url); }); globalThis.fetch = mockFetch; @@ -121,7 +179,9 @@ http://example.com/audio-seg1.m4s expect(state.selectedAudioTrackId).toBeDefined(); expect(owners.audioBufferActor).toBeDefined(); expect(owners.mediaSource).toBeDefined(); - expect(owners.mediaSource?.readyState).toBe('open'); + // readyState isn't asserted: with appendSegment mocked the stream completes + // instantly, so the MediaSource doesn't durably sit in 'open' (a created buffer + // actor implies addSourceBuffer ran, which requires an open MediaSource). }, { timeout: 2000 } ); @@ -163,7 +223,7 @@ http://example.com/audio-seg1.m4s throw new Error('Audio-only variant fetched the video media playlist'); } - return Promise.reject(new Error(`Unmocked URL: ${url}`)); + return unmockedFetchFallback(url); }); globalThis.fetch = mockFetch; @@ -184,7 +244,9 @@ http://example.com/audio-seg1.m4s expect(state.selectedAudioTrackId).toBeDefined(); expect(owners.audioBufferActor).toBeDefined(); expect(owners.mediaSource).toBeDefined(); - expect((owners.mediaSource as MediaSource).readyState).toBe('open'); + // readyState isn't asserted: with appendSegment mocked the stream completes + // instantly, so the MediaSource doesn't durably sit in 'open' (a created buffer + // actor implies addSourceBuffer ran, which requires an open MediaSource). // Video-side slots absent — no composed behavior in this variant // declares them. Behaviors that read these slots defensively @@ -236,7 +298,7 @@ http://example.com/audio-seg1.m4s throw new Error(`Audio-only variant fetched a non-audio playlist: ${url}`); } - return Promise.reject(new Error(`Unmocked URL: ${url}`)); + return unmockedFetchFallback(url); }); globalThis.fetch = mockFetch; diff --git a/packages/spf/src/playback/engines/hls/tests/engine.test.ts b/packages/spf/src/playback/engines/hls/tests/engine.test.ts index 1abf0bf0..ebd00cdd 100644 --- a/packages/spf/src/playback/engines/hls/tests/engine.test.ts +++ b/packages/spf/src/playback/engines/hls/tests/engine.test.ts @@ -8,15 +8,27 @@ vi.mock('../../../../media/dom/mse/append-segment', () => ({ appendSegment: vi.fn().mockResolvedValue(undefined), })); +// Fallback for URLs a test's mock doesn't handle explicitly. Segment/init +// requests resolve with an empty body — the appendSegment mock makes the bytes +// inert — so the failover monitor isn't tripped by unmocked segment fetches (a +// single failed fetch trips that CDN into cooldown, which empties the candidate +// set). Genuinely unknown URLs still reject loudly. +function unmockedFetchFallback(url: string): Promise { + // Non-empty body: `fetchStream` throws "Response has no body" on a null body + // (empty Uint8Array), which would itself trip the monitor. + if (/\.(m4s|mp4|ts|aac)(\?|$)/.test(url)) return Promise.resolve(new Response(new Uint8Array([0]))); + return Promise.reject(new Error(`Unmocked URL: ${url}`)); +} + describe('createSimpleHlsEngine', () => { let originalFetch: typeof globalThis.fetch; let consoleErrorSpy: ReturnType; - // Tests assert at actor-presence and state-shape level, not at "init - // segment appended" level — so unmocked init/segment URLs in the manifests - // are intentional. The fetch loop's reject path leaks a console.error in - // each test; suppress only the expected patterns so genuine failures still - // surface. + // Tests assert at actor-presence and state-shape level, not at "init segment + // appended" level. Audio/video segment fetches resolve via + // `unmockedFetchFallback` (inert under the appendSegment mock); text-track + // segment fetches still reject and leak a console.error. Suppress only the + // expected patterns so genuine failures still surface. const expectedErrorPatterns = [ /Unexpected error in segment loader.*Unmocked URL/s, /Failed to load text-track segment/, @@ -285,6 +297,85 @@ describe('createSimpleHlsEngine', () => { engine.destroy(); }); + it('codec-filters renditions via the injected canPlayTrack before selection', async () => { + const flush = () => Promise.resolve().then(() => Promise.resolve()); + // Reject HEVC; accept everything else. + const canPlayTrack = (track: { codecs?: string[] }) => !track.codecs?.some((c) => c.startsWith('hvc1')); + const engine = createSimpleHlsEngine({ canPlayTrack }); + + const videoTrack = (id: string, codec: string): PartiallyResolvedVideoTrack => ({ + type: 'video', + id, + codecs: [codec], + url: `https://example.com/${id}.m3u8`, + bandwidth: 4_800_000, + mimeType: 'video/mp4', + }); + + engine.state.presentation.set({ + id: 'pres-codec', + url: 'https://example.com/master.m3u8', + startTime: 0, + selectionSets: [ + { + id: 'v', + type: 'video', + switchingSets: [ + { + id: 'vs', + type: 'video', + tracks: [videoTrack('1080p-hevc', 'hvc1.1.6.L120.B0'), videoTrack('1080p-avc', 'avc1.640028')], + }, + ], + }, + ], + } as Presentation); + await flush(); + + // HEVC pruned upstream by the capability constraint; AVC selected. + expect(engine.state.selectedVideoTrackId.get()).toBe('1080p-avc'); + + engine.destroy(); + }); + + it('makes no video pick when no rendition is decodable', async () => { + const flush = () => Promise.resolve().then(() => Promise.resolve()); + const engine = createSimpleHlsEngine({ canPlayTrack: () => false }); + + engine.state.presentation.set({ + id: 'pres-unsupported', + url: 'https://example.com/master.m3u8', + startTime: 0, + selectionSets: [ + { + id: 'v', + type: 'video', + switchingSets: [ + { + id: 'vs', + type: 'video', + tracks: [ + { + type: 'video', + id: '1080p-hevc', + codecs: ['hvc1.1.6.L120.B0'], + url: 'https://example.com/1080p-hevc.m3u8', + bandwidth: 4_800_000, + mimeType: 'video/mp4', + } as PartiallyResolvedVideoTrack, + ], + }, + ], + }, + ], + } as Presentation); + await flush(); + + expect(engine.state.selectedVideoTrackId.get()).toBeUndefined(); + + engine.destroy(); + }); + it('auto-fails-over when a CDN fetch fails (monitor trips, failedCdns set)', async () => { const engine = createSimpleHlsEngine({ failover: { cooldownMs: 60_000 } }); @@ -455,7 +546,7 @@ http://example.com/segment1.m4s } // Fallback for unmocked URLs - return Promise.reject(new Error(`Unmocked URL: ${url}`)); + return unmockedFetchFallback(url); }); globalThis.fetch = mockFetch; @@ -523,7 +614,7 @@ http://example.com/audio-seg1.m4s ); } - return Promise.reject(new Error(`Unmocked URL: ${url}`)); + return unmockedFetchFallback(url); }); globalThis.fetch = mockFetch; @@ -571,9 +662,11 @@ http://example.com/audio-seg1.m4s // 4. MediaElement should be set expect(owners.mediaElement).toBe(mediaElement); - // 5. MediaSource should be created and open + // 5. MediaSource should be created expect(owners.mediaSource).toBeDefined(); - expect(owners.mediaSource?.readyState).toBe('open'); + // readyState isn't asserted: with appendSegment mocked the stream completes + // instantly, so the MediaSource doesn't durably sit in 'open' (a created buffer + // actor implies addSourceBuffer ran, which requires an open MediaSource). // 6. Video buffer cluster should be created (actor presence implies // `addSourceBuffer` ran; SourceBuffer itself is private to @@ -661,7 +754,7 @@ http://example.com/audio-b-seg1.m4s ); } - return Promise.reject(new Error(`Unmocked URL: ${url}`)); + return unmockedFetchFallback(url); }); globalThis.fetch = mockFetch; @@ -682,7 +775,9 @@ http://example.com/audio-b-seg1.m4s expect(state.presentation?.id).toBeDefined(); expect(state.selectedVideoTrackId).toBeDefined(); expect(state.selectedAudioTrackId).toBeDefined(); - expect(owners.mediaSource?.readyState).toBe('open'); + // readyState isn't asserted: with appendSegment mocked the stream completes + // instantly, so the MediaSource doesn't durably sit in 'open' (a created buffer + // actor implies addSourceBuffer ran, which requires an open MediaSource). expect(owners.videoBufferActor).toBeDefined(); expect(owners.audioBufferActor).toBeDefined(); }, @@ -715,7 +810,9 @@ http://example.com/audio-b-seg1.m4s expect(state.selectedAudioTrackId).toBeDefined(); // Fresh MediaSource + buffer actors (different instances from A) - expect(owners.mediaSource?.readyState).toBe('open'); + // readyState isn't asserted: with appendSegment mocked the stream completes + // instantly, so the MediaSource doesn't durably sit in 'open' (a created buffer + // actor implies addSourceBuffer ran, which requires an open MediaSource). expect(owners.mediaSource).not.toBe(sourceAMediaSource); expect(owners.videoBufferActor).not.toBe(sourceAVideoBufferActor); expect(owners.audioBufferActor).not.toBe(sourceAAudioBufferActor); @@ -752,7 +849,7 @@ http://example.com/video-seg1.m4s ); } - return Promise.reject(new Error(`Unmocked URL: ${url}`)); + return unmockedFetchFallback(url); }); globalThis.fetch = mockFetch; @@ -779,7 +876,9 @@ http://example.com/video-seg1.m4s // MediaSource should still be created expect(owners.mediaSource).toBeDefined(); - expect(owners.mediaSource?.readyState).toBe('open'); + // readyState isn't asserted: with appendSegment mocked the stream completes + // instantly, so the MediaSource doesn't durably sit in 'open' (a created buffer + // actor implies addSourceBuffer ran, which requires an open MediaSource). }, { timeout: 2000 } ); @@ -814,7 +913,7 @@ http://example.com/audio-seg1.m4s ); } - return Promise.reject(new Error(`Unmocked URL: ${url}`)); + return unmockedFetchFallback(url); }); globalThis.fetch = mockFetch; @@ -841,7 +940,9 @@ http://example.com/audio-seg1.m4s // MediaSource should still be created expect(owners.mediaSource).toBeDefined(); - expect(owners.mediaSource?.readyState).toBe('open'); + // readyState isn't asserted: with appendSegment mocked the stream completes + // instantly, so the MediaSource doesn't durably sit in 'open' (a created buffer + // actor implies addSourceBuffer ran, which requires an open MediaSource). }, { timeout: 2000 } ); @@ -876,7 +977,7 @@ http://example.com/video-seg1.m4s ); } - return Promise.reject(new Error(`Unmocked URL: ${url}`)); + return unmockedFetchFallback(url); }); globalThis.fetch = mockFetch; @@ -931,7 +1032,7 @@ http://example.com/video-seg1.m4s ); } - return Promise.reject(new Error(`Unmocked URL: ${url}`)); + return unmockedFetchFallback(url); }); globalThis.fetch = mockFetch; @@ -1008,7 +1109,7 @@ http://example.com/audio-seg1.m4s ); } - return Promise.reject(new Error(`Unmocked URL: ${url}`)); + return unmockedFetchFallback(url); }); globalThis.fetch = mockFetch; @@ -1044,7 +1145,9 @@ http://example.com/audio-seg1.m4s expect(state.selectedAudioTrackId).toBeDefined(); expect(owners.mediaSource).toBeDefined(); - expect(owners.mediaSource?.readyState).toBe('open'); + // readyState isn't asserted: with appendSegment mocked the stream completes + // instantly, so the MediaSource doesn't durably sit in 'open' (a created buffer + // actor implies addSourceBuffer ran, which requires an open MediaSource). expect(owners.videoBufferActor).toBeDefined(); expect(owners.audioBufferActor).toBeDefined(); }, @@ -1086,7 +1189,7 @@ http://example.com/seg1.m4s return Promise.resolve(new Response(new ArrayBuffer(100))); } - return Promise.reject(new Error(`Unmocked URL: ${url}`)); + return unmockedFetchFallback(url); }); globalThis.fetch = mockFetch; @@ -1159,7 +1262,7 @@ http://example.com/seg1.m4s ); } - return Promise.reject(new Error(`Unmocked URL: ${url}`)); + return unmockedFetchFallback(url); }); globalThis.fetch = mockFetch; @@ -1201,10 +1304,15 @@ http://example.com/seg1.m4s // The resolved track should be the selected one expect(resolvedTracks?.[0]?.id).toBe(state.selectedVideoTrackId); - // Should fetch: 1 multivariant + 1 media playlist + init attempt - // (Only selected quality, not all 3 qualities; init.mp4 is attempted but - // rejected by the mock — segment is not attempted since init fails first) - expect(mockFetch).toHaveBeenCalledTimes(3); + // The non-selected qualities are never resolved — only the selected track's + // media playlist is fetched. Asserts the intent directly rather than a brittle + // total fetch count (which shifts with init/segment loading of the selected track). + const fetchedUrls = mockFetch.mock.calls.map((call: unknown[]) => { + const input = call[0] as RequestInfo | URL; + return typeof input === 'string' ? input : input instanceof URL ? input.href : (input as Request).url; + }); + expect(fetchedUrls.some((u: string) => u.includes('video-720p.m3u8'))).toBe(false); + expect(fetchedUrls.some((u: string) => u.includes('video-1080p.m3u8'))).toBe(false); engine.destroy(); }); @@ -1259,7 +1367,7 @@ http://example.com/text-es-seg1.vtt ); } - return Promise.reject(new Error(`Unmocked URL: ${url}`)); + return unmockedFetchFallback(url); }); globalThis.fetch = mockFetch; @@ -1351,7 +1459,7 @@ http://example.com/text-es-seg1.vtt ); } - return Promise.reject(new Error(`Unmocked URL: ${url}`)); + return unmockedFetchFallback(url); }); globalThis.fetch = mockFetch; @@ -1427,7 +1535,7 @@ http://example.com/text-fr-seg1.vtt ); } - return Promise.reject(new Error(`Unmocked URL: ${url}`)); + return unmockedFetchFallback(url); }); globalThis.fetch = mockFetch; @@ -1512,7 +1620,7 @@ http://example.com/text-es-seg1.vtt ); } - return Promise.reject(new Error(`Unmocked URL: ${url}`)); + return unmockedFetchFallback(url); }); globalThis.fetch = mockFetch; @@ -1604,7 +1712,7 @@ http://example.com/video-seg1.m4s ); } - return Promise.reject(new Error(`Unmocked URL: ${url}`)); + return unmockedFetchFallback(url); }); globalThis.fetch = mockFetch; @@ -1698,7 +1806,7 @@ http://example.com/text-es-seg1.vtt ); } - return Promise.reject(new Error(`Unmocked URL: ${url}`)); + return unmockedFetchFallback(url); }); globalThis.fetch = mockFetch; @@ -1797,7 +1905,7 @@ http://example.com/seg2.m4s return Promise.resolve(new Response(new ArrayBuffer(1000))); } - return Promise.reject(new Error(`Unmocked URL: ${url}`)); + return unmockedFetchFallback(url); }); globalThis.fetch = mockFetch; @@ -1872,7 +1980,7 @@ http://example.com/audio-seg1.m4s return Promise.resolve(new Response(new ArrayBuffer(1000))); } - return Promise.reject(new Error(`Unmocked URL: ${url}`)); + return unmockedFetchFallback(url); }); globalThis.fetch = mockFetch;