mirror of
https://github.com/zoriya/v10.git
synced 2026-08-16 02:45:09 +00:00
feat(spf): capability probing (#1676)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Christian Pillsbury
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
b89f1e944c
commit
bce79ec424
@@ -1,7 +1,7 @@
|
||||
---
|
||||
status: draft
|
||||
status: partial
|
||||
date: 2026-05-20
|
||||
definition: technical
|
||||
definition: sketched
|
||||
---
|
||||
|
||||
# Capability probing
|
||||
@@ -25,18 +25,19 @@ DRM, and the unsupported-case error mapping.
|
||||
|
||||
## Status
|
||||
|
||||
- **Composition:** not implemented in `createSimpleHlsEngine`. A
|
||||
single late-failure check exists today: `isCodecSupported` in
|
||||
`media/dom/mse/mediasource-setup.ts` is called by `createSourceBuffer`
|
||||
inside `setupVideoBufferActors` / `setupAudioBufferActors` — by which
|
||||
point selection has already run and an unsupported variant may have
|
||||
been picked.
|
||||
- **Definition depth:** technical — scope and constraints articulated;
|
||||
no implementation. Source material: [SPF Epics Working Doc —
|
||||
candidate epics #17 (codec / container), #18 (multivariant CODECS),
|
||||
#19 (key-system)](https://www.notion.so/35f97a7f89d08123a13fecab1ca1cac4)
|
||||
(cluster D; each sized S; the Epics doc flags #17 + #18 as likely
|
||||
merging).
|
||||
- **Composition:** synchronous codec filtering is live in
|
||||
`createSimpleHlsEngine`. The `canPlayTrack` probe is injected into
|
||||
`track-switching`'s hard-constraints pre-pass (`excludeUnplayableTracks`),
|
||||
so undecodable renditions are pruned *before* selection. The pre-existing
|
||||
late-failure check (`isCodecSupported` inside `createSourceBuffer`) stays as
|
||||
a defensive backstop and should now rarely fire.
|
||||
- **Definition depth:** sketched — Phase 1 (probe primitive) and Phase 2
|
||||
(multivariant CODECS filtering) are implemented; key-system probing,
|
||||
`changeType()` probing, segment-level checking, no-playable surfacing
|
||||
(Phase 6), and Tier 2 overrides remain unimplemented. Source
|
||||
material: [SPF Epics Working Doc — candidate epics #17 (codec / container),
|
||||
#18 (multivariant CODECS),
|
||||
#19 (key-system)](https://www.notion.so/35f97a7f89d08123a13fecab1ca1cac4).
|
||||
- **Foundational** for cluster D — `[hevc-variant-selection]`,
|
||||
`[5.1-surround-selection]`, `[unsupported-case-error-mapping]`, and
|
||||
[drm-support](./drm-support.md) (GitHub issue #1411) all consume
|
||||
@@ -49,15 +50,15 @@ Scope slices around the capability-probing contract. Tier 1
|
||||
layer onto specific phases per the
|
||||
[Tier 1 / Tier 2 framing](./clusters.md#tier-1-spec-compliant-baseline-vs-tier-2-custom-behavior).
|
||||
|
||||
| Phase | What | Notes |
|
||||
| Phase | What | Status |
|
||||
|---|---|---|
|
||||
| Codec / container probing primitive | Uniform API wrapping `MediaSource.isTypeSupported` + `canPlayType`. Helpers for "given a `Track`, can we play it?" Builds on today's `isCodecSupported` | The minimum primitive everything else builds on. Today's `isCodecSupported` is the codec half; the wrapper formalizes the surface |
|
||||
| Multivariant CODECS-attribute filtering | At presentation resolution (post-parse), filter `presentation.selectionSets` to drop renditions whose `CODECS` doesn't decode on this browser. Filtered set is what selection behaviors operate over; unsupported renditions never reach selection | Tier 1 (spec-compliant). Today's late-failure path becomes a defensive fallback rarely exercised |
|
||||
| Media-playlist / segment-level capability checking | Per-segment CODECS verification + container detection at the media-playlist level. Catches mismatches the multivariant didn't declare | Tier 1. Largely defensive; expected to be rare for well-formed manifests |
|
||||
| Key-system capability probing | `requestMediaKeySystemAccess` for each candidate key system (Widevine, PlayReady, FairPlay, FairPlay-AirPlay). Returns supported configurations. **DRM-adjacent boundary:** this feature owns Tier 1 probing only; EME setup, license fetch, key delivery live under [drm-support](./drm-support.md) (GitHub issue #1411) | Async — pushes toward a new-behavior filter writer pattern rather than a derived signal |
|
||||
| Cross-codec transition (`changeType()`) probing | Probe whether `SourceBuffer.changeType()` is available, plus pair-wise support for specific codec transitions (AVC ↔ HEVC, AAC stereo ↔ AC-3 5.1, etc.). Browser support is fragile and pair-specific | Consumers decide whether to attempt mid-stream switches based on this probe; the `changeType()` call itself lives in those consumer features |
|
||||
| Unsupported-case error surfacing | When no candidate survives filtering, surface a clear error rather than failing late in `createSourceBuffer`. State-error slot or callback — the interface is defined here; consumer-side mapping lives in `[unsupported-case-error-mapping]` | The "fail loudly upstream" path |
|
||||
| Tier 2: customer probing overrides | Config-driven biases: "force AVC even when HEVC supported," "prefer hardware-backed DRM," "exclude codec X." Layered on top of Tier 1's spec-compliant filtering | Tier 2 (custom behavior). Often consumer-policy-driven |
|
||||
| Codec / container probing primitive | Uniform API wrapping `MediaSource.isTypeSupported` + `canPlayType`. Helpers for "given a `Track`, can we play it?" Builds on today's `isCodecSupported` | **Implemented** (codec half) — `canPlayTrack` in `media/dom/capabilities.ts`, memoized by MIME. `canPlayType` not wrapped yet |
|
||||
| Multivariant CODECS-attribute filtering | Post-parse, drop renditions whose `CODECS` doesn't decode on this browser, before selection. Filtered set is what selection behaviors operate over | **Implemented** — `excludeUnplayableTracks` constraint in `track-switching`'s pre-pass. Tier 1 (spec-compliant). The late-failure path is now a defensive fallback |
|
||||
| Media-playlist / segment-level capability checking | Per-segment CODECS verification + container detection at the media-playlist level. Catches mismatches the multivariant didn't declare | Not implemented. Tier 1; largely defensive, rare for well-formed manifests |
|
||||
| Key-system capability probing | `requestMediaKeySystemAccess` for each candidate key system (Widevine, PlayReady, FairPlay, FairPlay-AirPlay). Returns supported configurations. **DRM-adjacent boundary:** this feature owns Tier 1 probing only; EME setup, license fetch, key delivery live under [drm-support](./drm-support.md) (GitHub issue #1411) | Not implemented. Async — a slot-writer behavior, *not* the synchronous config-predicate route codec filtering took (see resolved open question) |
|
||||
| Cross-codec transition (`changeType()`) probing | Probe whether `SourceBuffer.changeType()` is available, plus pair-wise support for specific codec transitions (AVC ↔ HEVC, AAC stereo ↔ AC-3 5.1, etc.). Browser support is fragile and pair-specific | Not implemented. Consumers decide whether to attempt mid-stream switches based on this probe; the `changeType()` call itself lives in those consumer features |
|
||||
| Unsupported-case error surfacing | When no candidate survives filtering, surface a clear state rather than failing late in `createSourceBuffer` | Not implemented. A speculative per-type `noPlayable{Video,Audio}Tracks` flag was prototyped and **removed** (no consumer — write-only state + derivation stored as a slot). Today, when constraints prune a type that *has* tracks to empty, the behavior clears the selection (no pick) and `console.error`s as a placeholder; the late `createSourceBuffer` check is the backstop. Deferred in full to `[unsupported-case-error-mapping]`, which will replace the `console.error` with a surfaced shape (likely a derived `computed`, not a stored slot) once a consumer exists |
|
||||
| Tier 2: customer probing overrides | Config-driven biases: "force AVC even when HEVC supported," "prefer hardware-backed DRM," "exclude codec X." Layered on top of Tier 1's spec-compliant filtering | Not implemented. The `canPlayTrack` config injection point is the natural seam (override the default probe) |
|
||||
|
||||
## What's in scope vs out of scope
|
||||
|
||||
@@ -90,6 +91,42 @@ layer onto specific phases per the
|
||||
- Adapter / consumer-side error display
|
||||
- Consumer-specific error-code mappings (above-engine)
|
||||
|
||||
## Implementation surface
|
||||
|
||||
**Probe primitive:**
|
||||
|
||||
| Export | File | Role |
|
||||
|---|---|---|
|
||||
| `canPlayTrack` | `media/dom/capabilities.ts` | `(track) => boolean` — builds the MIME codec string and checks `MediaSource.isTypeSupported`, memoized by MIME. Unprobeable tracks (no `mimeType` / no `codecs`) pass through. **Asserts detected non-fMP4 containers (`video/mp2t`, `audio/aac`) unsupported without probing** — TS because the probe false-positives + no transmux; raw AAC because the pipeline assumes init segments (a temporary limitation — browser supports it) |
|
||||
| `CanPlayTrack` (type) | `media/types/index.ts` | DOM-free predicate type the constraint consumes; `canPlayTrack` is its DOM implementation |
|
||||
| Container detection (`CONTAINER_MIME_BY_EXTENSION`, `NON_FMP4_CONTAINER_MIMES`) | `media/hls/parse-media-playlist.ts` | Detects non-fMP4 containers per media playlist (no `#EXT-X-MAP` **and** a recognized segment extension: `.ts` → `video/mp2t`, `.aac` → `audio/aac`) and relabels the resolved track's `mimeType` from the fMP4 default. `canPlayTrack` then prunes them |
|
||||
| `applyContainerMimeType` | `media/utils/tracks.ts` | Propagates the detected container to every rendition of the **same type** (called from `resolve-track`): one resolved non-fMP4 playlist relabels all of that type's renditions, so the type is pruned from a single fetch. Scoped to one type — never crosses audio↔video (mixed-container sources exist), which also keeps per-type resolutions' writes disjoint (race-free) |
|
||||
|
||||
**Constraint (in `playback/behaviors/track-switching.ts`):**
|
||||
|
||||
| Piece | Role |
|
||||
|---|---|
|
||||
| `excludeUnplayableTracks` | Hard-constraint in the `applyConstraints` pre-pass; reads `config.canPlayTrack`, drops undecodable renditions before the rule chain. Shared by `switchVideoTrack` / `switchAudioTrack`, pooled with `excludeFailedCdns`. When it prunes a type that *has* tracks to empty, the selection clears (no pick) so a pick made before a relabel can't linger as unplayable |
|
||||
|
||||
**Engine wiring (`playback/engines/hls/engine.ts` + `engine-audio-only.ts`):**
|
||||
- `canPlayTrack` config — both engine factories default it to the DOM `canPlayTrack` in `finalConfig` (the audio-only variant too, so filtering isn't inert there); override to force-exclude a codec (the Tier 2 seam). Adapters forward it via `...config`.
|
||||
|
||||
**State slots:**
|
||||
- **Reads (constraint):** `presentation` candidates' `mimeType` + `codecs`, via `config.canPlayTrack`.
|
||||
- **Writes:** none — the constraint only narrows the candidate set the existing `selected*TrackId` writers pick from.
|
||||
|
||||
## Verification
|
||||
|
||||
- **Unit — `media/dom/tests/capabilities.test.ts`:** `canPlayTrack` returns the `isTypeSupported` verdict for a track's built MIME; memoizes per unique MIME (probes once); passes through (`true`) for unprobeable tracks (no `mimeType`, or empty/absent `codecs`); asserts non-fMP4 containers (`video/mp2t`, `audio/aac`) unsupported without consulting `isTypeSupported` (even with codecs).
|
||||
- **Unit — `media/hls/tests/parse-media-playlist.test.ts`:** relabels to `video/mp2t` / `audio/aac` when there's no `#EXT-X-MAP` and segments are `.ts` / `.aac` (query string ignored; `video/mp2t` for audio TS too); keeps the fMP4 default when an `#EXT-X-MAP` is present or the extension is unrecognized.
|
||||
- **Unit — `media/utils/tests/tracks.test.ts`:** `applyContainerMimeType` sets the MIME on every track of the given type, leaves other types untouched (never crosses audio↔video), idempotent.
|
||||
- **Unit — `playback/behaviors/tests/track-switching.test.ts`:** `excludeUnplayableTracks` prunes undecodable renditions before ranking (picks best playable codec); passes through with no probe wired; a user-selected unplayable track is still excluded (hard constraint beats the soft user filter); makes no pick when the constraint prunes every rendition from a cold start; **clears a prior pick when a later relabel prunes every rendition to empty** (the warm path — a pick made under the initial mp4 label, then the type relabeled to `video/mp2t`).
|
||||
- **Integration — `playback/engines/hls/tests/engine.test.ts`:** a mixed HEVC+AVC source with a `canPlayTrack` rejecting HEVC selects the AVC rendition; an all-undecodable source makes no video pick.
|
||||
|
||||
**Live smoke test:** verified in the SPF sandbox against the Apple `bipbop_4x3` stream (muxed-TS video + raw-`.aac` audio) — video relabels `video/mp2t`, audio relabels `audio/aac`, and both types are pruned to empty (per-type propagation: one fetch per type, not per rendition), so the selection clears (no pick) instead of leaving the initial mp4-labeled pick selected to stall deep in the pipeline. An audio-only `.aac` source likewise makes no pick.
|
||||
|
||||
**Out of scope / deferred:** `canPlayType` wrapper, key-system probing, `changeType()` probing, per-segment CODECS checking, no-playable surfacing / the full error-code interface, and Tier 2 override config. Container *detection* covers MPEG-TS + raw ADTS AAC (not `.mp3` etc. yet); both are asserted **unplayable** for now. *Playing* them is separate follow-up work — TS needs a transmux pipeline; **raw AAC is genuinely browser-supported (Chrome/Safari) and could be played by removing the pipeline's init-segment assumption** (the segment loader queues an `append-init` with an empty URL, and append handling is fMP4-shaped).
|
||||
|
||||
## Likely cross-cutting impact
|
||||
|
||||
- **Selection behaviors** — `selectVideoTrack` / `switchVideoQuality`
|
||||
@@ -115,30 +152,85 @@ layer onto specific phases per the
|
||||
`changeType()` call lives in their own feature work; this feature
|
||||
provides the "can we?" answer.
|
||||
|
||||
## Resolved during Phase 1–2 implementation
|
||||
|
||||
- **Filter-writer pattern → split (not unified).** Synchronous codec
|
||||
probing is a pure function of (track, environment), so it landed as a
|
||||
**config-injected predicate** (`canPlayTrack`) read by the
|
||||
`excludeUnplayableTracks` hard constraint — no behavior, no state slot.
|
||||
This mirrors `getCdnId` injection, not the `failedCdns ←
|
||||
setupFailoverMonitor` slot-writer (which exists because CDN cooldown is
|
||||
*dynamic*; codec support is static). Async key-system probing (Phase 4)
|
||||
will take the slot-writer route, coexisting in the same pre-pass — the
|
||||
two are genuinely split by sync-vs-async.
|
||||
- **Cache-eager vs lazy → lazy + memoized.** The constraint probes each
|
||||
candidate at apply time; `canPlayTrack` memoizes by built MIME string, so
|
||||
each unique MIME is asked once without an upfront sweep.
|
||||
- **No-playable surfacing was removed (not deferred-as-built).** A per-type
|
||||
`noPlayable*` flag was prototyped — set when a type's candidates pruned to
|
||||
empty (codec filtering *or* CDN-failover cooldown). It was deleted: nothing
|
||||
consumed it (write-only state), and it stored a derivation in a slot an
|
||||
effect wrote rather than a `computed`. Today an emptied type that *had* tracks
|
||||
clears the selection (no pick) and `console.error`s as a placeholder; the late
|
||||
`createSourceBuffer` check is the backstop, and a track with no declared
|
||||
`CODECS` (optional per spec) is unprobeable and passes through. The
|
||||
*had-candidates-vs-never-had-candidates* distinction a bare "candidate set is
|
||||
empty" check can't make (a video-only source's absent audio is empty but not
|
||||
an error) is **already** made — the clear/error is gated on a non-zero
|
||||
pre-constraint track count for the type. When a consumer materializes,
|
||||
`[unsupported-case-error-mapping]` replaces the `console.error` with the
|
||||
surfaced shape (likely a `computed`).
|
||||
- **Container-detection scope → non-fMP4 detection (TS + raw AAC), per-track-type,
|
||||
marked unplayable.** The media-playlist parser relabels a resolved non-fMP4
|
||||
rendition (no `#EXT-X-MAP` + a recognized extension: `.ts` → `video/mp2t`,
|
||||
`.aac` → `audio/aac`), and `resolve-track` propagates that MIME to every
|
||||
rendition of the **same type** (`applyContainerMimeType`), so a type is pruned
|
||||
from a single resolved playlist. Propagation is **same-type, not cross-type**:
|
||||
Apple `bipbop_4x3` is mixed-container (muxed-TS video + raw-`.aac` audio), so a
|
||||
cross-type "whole source is one container" assumption was both *wrong* for that
|
||||
stream and racy under concurrent per-type resolution; same-type is safe (an ABR
|
||||
ladder shares its container) and race-free (disjoint writes). Both are asserted
|
||||
unplayable, for different reasons: TS because `isTypeSupported('video/mp2t…')`
|
||||
false-positives on Chromium *and* there's no transmux pipeline; raw AAC as a
|
||||
**temporary** limitation — the browser genuinely supports it (Chrome/Safari
|
||||
decode bare `audio/aac`; Firefox doesn't), but our segment loader / append
|
||||
pipeline assumes an `EXT-X-MAP` init segment, so it would fetch-but-never-buffer
|
||||
(a silent stall) today. Making raw AAC playable (remove the init-segment
|
||||
assumption; bare-MIME probe + projection) is deliberately out of scope here —
|
||||
see [container-support](./container-support.md).
|
||||
|
||||
## Open questions
|
||||
|
||||
- **Filter-writer pattern.** New behavior vs derived signal? Async
|
||||
key-system probing pushes toward new-behavior (writes a filter
|
||||
slot after async resolves). Synchronous codec probing could be a
|
||||
derived signal. Unified or split?
|
||||
- **Cache-eager vs lazy probing.** Probe everything upfront (simpler,
|
||||
worst-case cost) vs probe-on-demand (more efficient for sources
|
||||
with many renditions). Affects state-slot writer pattern.
|
||||
- **Surfacing "nothing playable" as a (fatal) error.** The pre-pass can prune a
|
||||
type to empty — every rendition undecodable (fatal: the source can't play at
|
||||
all) or, transiently, every CDN cooled down (recoverable). Today both clear
|
||||
the selection (no pick) and `console.error` — *uniformly*, without yet
|
||||
distinguishing fatal from recoverable; the late `createSourceBuffer` check
|
||||
remains the backstop. `[unsupported-case-error-mapping]` will want this
|
||||
surfaced. Open: a derived
|
||||
`computed` read by that consumer (preferred — see the removed `noPlayable*`
|
||||
note under "Resolved"), or a dedicated **behavior** if surfacing needs to own
|
||||
emit/clear, the fatal-vs-recoverable classification, and the
|
||||
had-vs-never-had-candidates gating? Lean `computed` unless that
|
||||
lifecycle/coordination forces a behavior. Whatever lands must distinguish
|
||||
fatal (no decodable rendition) from recoverable (failover cooldown) so the
|
||||
consumer doesn't fire a terminal error on a transient condition.
|
||||
- **Sharing the constraint pre-pass with user-selection logic.** Constraints
|
||||
(`excludeUnplayableTracks`, `excludeFailedCdns`) live *inside*
|
||||
`track-switching`'s pre-pass, invisible outside it. A hard constraint already
|
||||
correctly beats a user pick (tested — the user's unplayable selection is
|
||||
pruned before `filterByUserSelection` runs), but the override is *silent* and
|
||||
a user-facing track list can still present a rendition that will never play.
|
||||
Open: expose the post-constraint candidate set (per type) as shared state for
|
||||
selection UIs / the error surface to consume, or factor the constraint list
|
||||
into a reusable unit those surfaces apply themselves? Same "expose the
|
||||
candidate set" thread as the `noPlayable*` removal.
|
||||
- **`changeType()` pair-wise probing API.** Probe all pairs upfront,
|
||||
probe lazily on switch attempt, or expose a `canChangeType(from,
|
||||
to)` predicate that callers invoke?
|
||||
- **Tier 2 customer override surface.** Config-driven (engine-wide)
|
||||
vs per-source vs both? Per-source is more flexible but harder to
|
||||
wire.
|
||||
- **Container-detection scope.** [container-support](./container-support.md)
|
||||
is documented as standalone (cluster-less; MSE doesn't accept
|
||||
non-fMP4 containers per spec, so the concern is structurally
|
||||
different from capability-probing's "what can the browser decode
|
||||
in fMP4?" framing). Open: should this feature surface container-
|
||||
format detection as part of multivariant filtering (filter out
|
||||
MPEG-TS variants when no transmuxer is composed)? That would
|
||||
resolve the scope intersection cleanly without building the
|
||||
transmuxer.
|
||||
|
||||
## Related features
|
||||
|
||||
@@ -161,12 +253,12 @@ layer onto specific phases per the
|
||||
consumer-facing error mapping on top of this feature's error
|
||||
primitive.
|
||||
- **[container-support](./container-support.md)** — standalone
|
||||
feature (cluster-less); resolved per the doc's framing that MSE
|
||||
doesn't accept non-fMP4 containers per spec, making the concern
|
||||
fundamentally different from capability-probing's framing.
|
||||
Container-detection-without-transmuxer remains a possible cross-
|
||||
feature integration point (filter MPEG-TS variants when no
|
||||
transmuxer is composed).
|
||||
feature (cluster-less) for *playing* non-fMP4 containers. The
|
||||
detection half landed here: MPEG-TS (`.ts` → `video/mp2t`) and raw
|
||||
ADTS AAC (`.aac` → `audio/aac`) are detected and asserted unplayable.
|
||||
Playing them lives there — TS needs a transmux pipeline; raw AAC just
|
||||
needs the init-segment assumption removed from the segment loader /
|
||||
append pipeline (the browser already supports it).
|
||||
|
||||
## See also
|
||||
|
||||
|
||||
Reference in New Issue
Block a user