mirror of https://github.com/zoriya/v10.git synced 2026-08-12 17:09:55 +00:00

Files

T

bce79ec424 feat(spf): capability probing (#1676 )

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

2026-06-17 11:22:49 -07:00

21 KiB

Raw Blame History

status, date, definition

status	date	definition
partial	2026-05-20	sketched

Capability probing

The engine's foundation for determining what the browser can actually play: codec / container support via MediaSource.isTypeSupported and canPlayType, key-system support via requestMediaKeySystemAccess, and SourceBuffer.changeType() availability for mid-stream codec transitions. Filters the candidate set before selection, so unsupported renditions don't survive into the pipeline and fail late.

A Media-src feature in the framing from clusters.md § Feature classification axes: without it, sources with browser-incompatible variants fail late at createSourceBuffer instead of failing gracefully (or falling back to a compatible variant). Cluster D foundation; the clusters.md § Capability probing section flags this as "deserves a dedicated home" with high reuse — consumers include HEVC variant selection, 5.1 surround selection, DRM, and the unsupported-case error mapping.

Status

Composition: synchronous codec filtering is live in createSimpleHlsEngine. The canPlayTrack probe is injected into track-switching's hard-constraints pre-pass (excludeUnplayableTracks), so undecodable renditions are pruned before selection. The pre-existing late-failure check (isCodecSupported inside createSourceBuffer) stays as a defensive backstop and should now rarely fire.
Definition depth: sketched — Phase 1 (probe primitive) and Phase 2 (multivariant CODECS filtering) are implemented; key-system probing, changeType() probing, segment-level checking, no-playable surfacing (Phase 6), and Tier 2 overrides remain unimplemented. Source material: SPF Epics Working Doc — candidate epics #17 (codec / container), #18 (multivariant CODECS), #19 (key-system).
Foundational for cluster D — [hevc-variant-selection], [5.1-surround-selection], [unsupported-case-error-mapping], and drm-support (GitHub issue #1411) all consume probing output.

Phases of complexity

Scope slices around the capability-probing contract. Tier 1 (spec-compliant filtering) and Tier 2 (customer-driven overrides) layer onto specific phases per the Tier 1 / Tier 2 framing.

Phase	What	Status
Codec / container probing primitive	Uniform API wrapping `MediaSource.isTypeSupported` + `canPlayType`. Helpers for "given a `Track`, can we play it?" Builds on today's `isCodecSupported`	Implemented (codec half) — `canPlayTrack` in `media/dom/capabilities.ts`, memoized by MIME. `canPlayType` not wrapped yet
Multivariant CODECS-attribute filtering	Post-parse, drop renditions whose `CODECS` doesn't decode on this browser, before selection. Filtered set is what selection behaviors operate over	Implemented — `excludeUnplayableTracks` constraint in `track-switching`'s pre-pass. Tier 1 (spec-compliant). The late-failure path is now a defensive fallback
Media-playlist / segment-level capability checking	Per-segment CODECS verification + container detection at the media-playlist level. Catches mismatches the multivariant didn't declare	Not implemented. Tier 1; largely defensive, rare for well-formed manifests
Key-system capability probing	`requestMediaKeySystemAccess` for each candidate key system (Widevine, PlayReady, FairPlay, FairPlay-AirPlay). Returns supported configurations. DRM-adjacent boundary: this feature owns Tier 1 probing only; EME setup, license fetch, key delivery live under drm-support (GitHub issue #1411)	Not implemented. Async — a slot-writer behavior, not the synchronous config-predicate route codec filtering took (see resolved open question)
Cross-codec transition (`changeType()`) probing	Probe whether `SourceBuffer.changeType()` is available, plus pair-wise support for specific codec transitions (AVC ↔ HEVC, AAC stereo ↔ AC-3 5.1, etc.). Browser support is fragile and pair-specific	Not implemented. Consumers decide whether to attempt mid-stream switches based on this probe; the `changeType()` call itself lives in those consumer features
Unsupported-case error surfacing	When no candidate survives filtering, surface a clear state rather than failing late in `createSourceBuffer`	Not implemented. A speculative per-type `noPlayable{Video,Audio}Tracks` flag was prototyped and removed (no consumer — write-only state + derivation stored as a slot). Today, when constraints prune a type that has tracks to empty, the behavior clears the selection (no pick) and `console.error`s as a placeholder; the late `createSourceBuffer` check is the backstop. Deferred in full to `[unsupported-case-error-mapping]`, which will replace the `console.error` with a surfaced shape (likely a derived `computed`, not a stored slot) once a consumer exists
Tier 2: customer probing overrides	Config-driven biases: "force AVC even when HEVC supported," "prefer hardware-backed DRM," "exclude codec X." Layered on top of Tier 1's spec-compliant filtering	Not implemented. The `canPlayTrack` config injection point is the natural seam (override the default probe)

What's in scope vs out of scope

In scope:

All phases above
Browser-API wrappers (isTypeSupported, canPlayType, requestMediaKeySystemAccess, changeType() availability)
Filter-writer behavior + the filtered-candidate-set slot pattern
Error-surfacing primitive (used by both this feature and [unsupported-case-error-mapping])

Out of scope (separate Media-src candidate features):

hevc-variant-selection — consumer. Uses Tier 1 (select HEVC if supported, fallback to AVC)
- cross-codec transition probing (mid-stream switching).
5.1-surround-selection — consumer. Same shape as HEVC, on the audio channel-count axis. Adds a 5.1-specific runtime-detection phase (downstream-environment- aware channel preference) with no HEVC analog.
drm-support (GitHub issue #1411) — EME setup, license handling. This feature owns the "what key systems are available?" probe; drm-support uses that answer to set up keys.
Multi-language-audio Tier 2 mid-stream codec switch — consumer of changeType() probing.
[unsupported-case-error-mapping] — sister feature; maps the error-surfacing primitive to consumer-facing codes / messages.

Out of scope (different architectural layer):

Adapter / consumer-side error display
Consumer-specific error-code mappings (above-engine)

Implementation surface

Probe primitive:

Export	File	Role
`canPlayTrack`	`media/dom/capabilities.ts`	`(track) => boolean` — builds the MIME codec string and checks `MediaSource.isTypeSupported`, memoized by MIME. Unprobeable tracks (no `mimeType` / no `codecs`) pass through. Asserts detected non-fMP4 containers (`video/mp2t`, `audio/aac`) unsupported without probing — TS because the probe false-positives + no transmux; raw AAC because the pipeline assumes init segments (a temporary limitation — browser supports it)
`CanPlayTrack` (type)	`media/types/index.ts`	DOM-free predicate type the constraint consumes; `canPlayTrack` is its DOM implementation
Container detection (`CONTAINER_MIME_BY_EXTENSION`, `NON_FMP4_CONTAINER_MIMES`)	`media/hls/parse-media-playlist.ts`	Detects non-fMP4 containers per media playlist (no `#EXT-X-MAP` and a recognized segment extension: `.ts` → `video/mp2t`, `.aac` → `audio/aac`) and relabels the resolved track's `mimeType` from the fMP4 default. `canPlayTrack` then prunes them
`applyContainerMimeType`	`media/utils/tracks.ts`	Propagates the detected container to every rendition of the same type (called from `resolve-track`): one resolved non-fMP4 playlist relabels all of that type's renditions, so the type is pruned from a single fetch. Scoped to one type — never crosses audio↔video (mixed-container sources exist), which also keeps per-type resolutions' writes disjoint (race-free)

Constraint (in playback/behaviors/track-switching.ts):

Piece	Role
`excludeUnplayableTracks`	Hard-constraint in the `applyConstraints` pre-pass; reads `config.canPlayTrack`, drops undecodable renditions before the rule chain. Shared by `switchVideoTrack` / `switchAudioTrack`, pooled with `excludeFailedCdns`. When it prunes a type that has tracks to empty, the selection clears (no pick) so a pick made before a relabel can't linger as unplayable

Engine wiring (playback/engines/hls/engine.ts + engine-audio-only.ts):

canPlayTrack config — both engine factories default it to the DOM canPlayTrack in finalConfig (the audio-only variant too, so filtering isn't inert there); override to force-exclude a codec (the Tier 2 seam). Adapters forward it via ...config.

State slots:

Reads (constraint): presentation candidates' mimeType + codecs, via config.canPlayTrack.
Writes: none — the constraint only narrows the candidate set the existing selected*TrackId writers pick from.

Verification

Unit — media/dom/tests/capabilities.test.ts: canPlayTrack returns the isTypeSupported verdict for a track's built MIME; memoizes per unique MIME (probes once); passes through (true) for unprobeable tracks (no mimeType, or empty/absent codecs); asserts non-fMP4 containers (video/mp2t, audio/aac) unsupported without consulting isTypeSupported (even with codecs).
Unit — media/hls/tests/parse-media-playlist.test.ts: relabels to video/mp2t / audio/aac when there's no #EXT-X-MAP and segments are .ts / .aac (query string ignored; video/mp2t for audio TS too); keeps the fMP4 default when an #EXT-X-MAP is present or the extension is unrecognized.
Unit — media/utils/tests/tracks.test.ts: applyContainerMimeType sets the MIME on every track of the given type, leaves other types untouched (never crosses audio↔video), idempotent.
Unit — playback/behaviors/tests/track-switching.test.ts: excludeUnplayableTracks prunes undecodable renditions before ranking (picks best playable codec); passes through with no probe wired; a user-selected unplayable track is still excluded (hard constraint beats the soft user filter); makes no pick when the constraint prunes every rendition from a cold start; clears a prior pick when a later relabel prunes every rendition to empty (the warm path — a pick made under the initial mp4 label, then the type relabeled to video/mp2t).
Integration — playback/engines/hls/tests/engine.test.ts: a mixed HEVC+AVC source with a canPlayTrack rejecting HEVC selects the AVC rendition; an all-undecodable source makes no video pick.

Live smoke test: verified in the SPF sandbox against the Apple bipbop_4x3 stream (muxed-TS video + raw-.aac audio) — video relabels video/mp2t, audio relabels audio/aac, and both types are pruned to empty (per-type propagation: one fetch per type, not per rendition), so the selection clears (no pick) instead of leaving the initial mp4-labeled pick selected to stall deep in the pipeline. An audio-only .aac source likewise makes no pick.

Out of scope / deferred: canPlayType wrapper, key-system probing, changeType() probing, per-segment CODECS checking, no-playable surfacing / the full error-code interface, and Tier 2 override config. Container detection covers MPEG-TS + raw ADTS AAC (not .mp3 etc. yet); both are asserted unplayable for now. Playing them is separate follow-up work — TS needs a transmux pipeline; raw AAC is genuinely browser-supported (Chrome/Safari) and could be played by removing the pipeline's init-segment assumption (the segment loader queues an append-init with an empty URL, and append handling is fMP4-shaped).

Likely cross-cutting impact

Selection behaviors — selectVideoTrack / switchVideoQuality / selectAudioTrack / selectTextTrack pickers read filtered set, not raw presentation.selectionSets. Same shape as the userVideoTrackSelection constraint pattern in video-abr.
mse-mms-pipeline late-failure path — createSourceBuffer's throw on unsupported codec becomes a defensive fallback. With upstream filtering it should rarely fire; the throw stays as a structural guarantee.
presentation-modeling — parser stays format-neutral. Capability filtering is post-parse, before selection. The architectural doc's parser-interface contract isn't affected.
video-abr — selectQuality operates over the filtered set. No code change in ABR itself; just narrower input.
DRM gate — key-system probing becomes the first DRM gate. Subsequent EME setup, license fetch, key delivery happen under drm-support, gated on probing's verdict. Crisp boundary: probing answers "can we?"; DRM-support answers "set it up."
Cross-codec transition consumers — HEVC, 5.1, multi-language- audio Tier 2 all need changeType() probing. The actual changeType() call lives in their own feature work; this feature provides the "can we?" answer.

Resolved during Phase 1–2 implementation

Filter-writer pattern → split (not unified). Synchronous codec probing is a pure function of (track, environment), so it landed as a config-injected predicate (canPlayTrack) read by the excludeUnplayableTracks hard constraint — no behavior, no state slot. This mirrors getCdnId injection, not the failedCdns ← setupFailoverMonitor slot-writer (which exists because CDN cooldown is dynamic; codec support is static). Async key-system probing (Phase 4) will take the slot-writer route, coexisting in the same pre-pass — the two are genuinely split by sync-vs-async.
Cache-eager vs lazy → lazy + memoized. The constraint probes each candidate at apply time; canPlayTrack memoizes by built MIME string, so each unique MIME is asked once without an upfront sweep.
No-playable surfacing was removed (not deferred-as-built). A per-type noPlayable* flag was prototyped — set when a type's candidates pruned to empty (codec filtering or CDN-failover cooldown). It was deleted: nothing consumed it (write-only state), and it stored a derivation in a slot an effect wrote rather than a computed. Today an emptied type that had tracks clears the selection (no pick) and console.errors as a placeholder; the late createSourceBuffer check is the backstop, and a track with no declared CODECS (optional per spec) is unprobeable and passes through. The had-candidates-vs-never-had-candidates distinction a bare "candidate set is empty" check can't make (a video-only source's absent audio is empty but not an error) is already made — the clear/error is gated on a non-zero pre-constraint track count for the type. When a consumer materializes, [unsupported-case-error-mapping] replaces the console.error with the surfaced shape (likely a computed).
Container-detection scope → non-fMP4 detection (TS + raw AAC), per-track-type, marked unplayable. The media-playlist parser relabels a resolved non-fMP4 rendition (no #EXT-X-MAP + a recognized extension: .ts → video/mp2t, .aac → audio/aac), and resolve-track propagates that MIME to every rendition of the same type (applyContainerMimeType), so a type is pruned from a single resolved playlist. Propagation is same-type, not cross-type: Apple bipbop_4x3 is mixed-container (muxed-TS video + raw-.aac audio), so a cross-type "whole source is one container" assumption was both wrong for that stream and racy under concurrent per-type resolution; same-type is safe (an ABR ladder shares its container) and race-free (disjoint writes). Both are asserted unplayable, for different reasons: TS because isTypeSupported('video/mp2t…') false-positives on Chromium and there's no transmux pipeline; raw AAC as a temporary limitation — the browser genuinely supports it (Chrome/Safari decode bare audio/aac; Firefox doesn't), but our segment loader / append pipeline assumes an EXT-X-MAP init segment, so it would fetch-but-never-buffer (a silent stall) today. Making raw AAC playable (remove the init-segment assumption; bare-MIME probe + projection) is deliberately out of scope here — see container-support.

Open questions

Surfacing "nothing playable" as a (fatal) error. The pre-pass can prune a type to empty — every rendition undecodable (fatal: the source can't play at all) or, transiently, every CDN cooled down (recoverable). Today both clear the selection (no pick) and console.error — uniformly, without yet distinguishing fatal from recoverable; the late createSourceBuffer check remains the backstop. [unsupported-case-error-mapping] will want this surfaced. Open: a derived computed read by that consumer (preferred — see the removed noPlayable* note under "Resolved"), or a dedicated behavior if surfacing needs to own emit/clear, the fatal-vs-recoverable classification, and the had-vs-never-had-candidates gating? Lean computed unless that lifecycle/coordination forces a behavior. Whatever lands must distinguish fatal (no decodable rendition) from recoverable (failover cooldown) so the consumer doesn't fire a terminal error on a transient condition.
Sharing the constraint pre-pass with user-selection logic. Constraints (excludeUnplayableTracks, excludeFailedCdns) live inside track-switching's pre-pass, invisible outside it. A hard constraint already correctly beats a user pick (tested — the user's unplayable selection is pruned before filterByUserSelection runs), but the override is silent and a user-facing track list can still present a rendition that will never play. Open: expose the post-constraint candidate set (per type) as shared state for selection UIs / the error surface to consume, or factor the constraint list into a reusable unit those surfaces apply themselves? Same "expose the candidate set" thread as the noPlayable* removal.
changeType() pair-wise probing API. Probe all pairs upfront, probe lazily on switch attempt, or expose a canChangeType(from, to) predicate that callers invoke?
Tier 2 customer override surface. Config-driven (engine-wide) vs per-source vs both? Per-source is more flexible but harder to wire.

mse-mms-pipeline — owns isCodecSupported + the late-failure path. This feature builds on those and moves the check upstream.
presentation-modeling (architectural) — data shape this feature filters; parser stays format-neutral.
video-abr — quality selection narrowed to filtered candidates.
audio-playback / subtitles — per-type selection over filtered candidates.
multi-language-audio (coarse) — Tier 2 mid-stream codec switch consumes changeType() probing.
hevc-variant-selection — consumer; selection + cross-codec switching.
5.1-surround-selection — consumer.
drm-support (GitHub issue #1411) — owns EME + license; consumes key-system probing.
[unsupported-case-error-mapping] (candidate) — sister; consumer-facing error mapping on top of this feature's error primitive.
container-support — standalone feature (cluster-less) for playing non-fMP4 containers. The detection half landed here: MPEG-TS (.ts → video/mp2t) and raw ADTS AAC (.aac → audio/aac) are detected and asserted unplayable. Playing them lives there — TS needs a transmux pipeline; raw AAC just needs the init-segment assumption removed from the segment loader / append pipeline (the browser already supports it).

21 KiB Raw Blame History Unescape Escape