mirror of https://github.com/zoriya/v10.git synced 2026-08-16 02:45:09 +00:00

Files

T

c9d6ccecdd docs(spf): seed feature registry, clusters reference, and document-feature skill (#1581 )

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-21 12:07:33 -07:00

16 KiB

Raw Blame History

status, date, definition

status	date	definition
draft	2026-05-20	technical

Multi-signal ABR

Extension to the existing video and audio ABR algorithm to incorporate non-bandwidth signals: network type / Save-Data preference, thermal / decode-pressure inference, battery state, viewport visibility, and customer-facing quality-preference policy. Today's selectQuality takes (candidates, bandwidth, config) → track; this feature widens the input axis so selection responds to more than just throughput.

A Media-src feature at the baseline (sensing-driven biases that make playback more appropriate to device + network conditions) layered with Player feature at the customer-preference tier (additive config-driven biases). Cluster C member, sibling to video-abr and audio-abr. Distinct from rendition-selection-caps: caps narrow the candidate set before selection; multi-signal-ABR extends the selection algorithm itself.

Status

Composition: not implemented in createSimpleHlsEngine. Today switchVideoQuality (and the upcoming switchAudioQuality per audio-abr) read only bandwidthState for the algorithm input.
Definition depth: technical — per-signal phases articulable from known browser APIs and the existing video-abr precedent; signal-fusion algorithm shape is open.
Prerequisite chain: depends on video-abr (and optionally audio-abr) for the base algorithm being extended. Independent of rendition-selection-caps — caps and signals are complementary, not blocking.

Phases of complexity

Content phases by signal type. Each row except the last is a discrete signal-source unit; the final "Signal fusion" row covers the algorithm-extension work that consumes them.

Phase	Signal source	Mechanism	Notes
Network-type / Save-Data awareness	`navigator.connection.effectiveType` (`'slow-2g'` / `'2g'` / `'3g'` / `'4g'`) and `navigator.connection.saveData` boolean. `connection.change` event for mid-playback transitions	Middle-pattern monitor behavior writes a `networkConditions` state slot; ABR algorithm reads as a downward bias factor (e.g., `'slow-2g'`/`'2g'` aggressively bias down; `saveData=true` similar). Naive: read once at startup; full: react to `connection.change`	Browser support: Chrome / Edge / Firefox have Network Information API; Safari does not expose it. Applies to both video and audio ABR (network conditions affect both equally). Available signal is coarse-grained but covers the high-value cases (cellular detection)
Thermal / decode-pressure awareness	Derived from `VideoPlaybackQuality.droppedVideoFrames` delta over a rolling window. Possibly supplemented by `PerformanceObserver` long-task durations as a CPU-pressure proxy. No direct browser thermal API	Middle-pattern monitor behavior samples dropped-frames periodically; computes a rolling drop rate; writes `thermalPressure` (or `decodePressure`) state slot; ABR reads as a downward bias when sustained drops exceed a threshold	Heuristic-only; the hardest signal to detect reliably. Naive: dropped-frames threshold + downgrade. Full: combine dropped-frames + long-task signals. Video-ABR primarily; audio-ABR rarely thermally-pressured (audio decode is cheap). One open question: distinguishing decode-pressure (real thermal/CPU issue) from network-driven gaps (which dropped-frames doesn't directly distinguish from underflow)
Battery awareness	`navigator.getBattery()` returning `{ level, charging, chargingTime, dischargingTime }` + `levelchange` / `chargingchange` events	Middle-pattern monitor behavior subscribes to battery events; writes `batteryState` state slot; ABR reads with policy like "when `level < 0.2 && !charging`, bias toward lower bitrate to extend battery"	Browser support: Chrome desktop yes; Firefox removed (security/fingerprinting concerns); Safari no. Treat as best-effort enhancement — when `getBattery` unavailable, slot stays `undefined` and bias is skipped. Applies to both video and audio ABR but video bias is more impactful (higher decode + GPU cost)
Visibility awareness	`document.visibilityState` (`'visible'` / `'hidden'`) + `IntersectionObserver` on the video element for in-page off-screen detection. Possibly Picture-in-Picture state via `navigator.mediaSession` or `document.pictureInPictureElement` for distinguishing "hidden tab" from "PiP-visible"	Middle-pattern monitor behavior tracks both signals; writes `playerVisibility` state slot (e.g., `'visible'` / `'hidden'` / `'pip'` / `'off-screen'`); ABR reads: pause aggressive upgrades when not visible; consider downgrade for prolonged hidden state	Mostly video-applicable (audio plays through headphones regardless of visual visibility). PiP is a critical case: tab can be hidden but PiP is visible to user, so don't downgrade. Cross-cutting with the broader engine — segment-loading might also want to pause forward-buffer fetching when hidden, but that's buffer-management's concern, not this feature's
Customer-preference policy	Config-driven `qualityPreference: 'auto' \| 'data-saver' \| 'best-quality' \| 'specific-level'` or similar	No sensing; config slot read by ABR as a bias-strength scalar. `'data-saver'` applies a strong downward bias; `'best-quality'` applies an upward bias respecting safety margins; `'auto'` is the default neutral behavior	Pure config / Player-feature-flavored. Customer policy reigns over automatic biases (e.g., explicit `'best-quality'` overrides battery-low downward bias, but never overrides bandwidth-safety floor). Applies to both video and audio ABR. Adapter-level customer-facing UI ("Auto" / "Data saver" toggles) consumes this config
Signal fusion / algorithm extension	Integration phase: combine bandwidth + per-signal biases into a single `selectQuality` decision. Today's `selectQuality(candidates, bandwidth, config)` becomes `selectQuality(candidates, bandwidth, signals, config)` — signals is a struct of optional inputs from the five sources above	Algorithm extension. Open question on weighting strategy: bias-factor weighting (each signal produces a multiplier on the effective bandwidth) vs decision-tree (signals narrow / shift candidate set) vs scoring-function (each candidate scored against all inputs, highest score wins)	The hard algorithmic part. Bandwidth supremacy: non-bandwidth biases must not override the bandwidth safety floor (`track.bandwidth ≤ measuredBandwidth × safetyMargin`). Applies uniformly to video and audio ABR — the algorithm extension is signal-shape-agnostic

What's in scope vs out of scope

In scope:

All six phases above for HLS multi-bitrate sources (video and audio variants where ABR is enabled)
Per-signal middle-pattern monitor behaviors with graceful no-op when the underlying browser API is unavailable
New state slots for each signal: networkConditions, thermalPressure, batteryState, playerVisibility, customerQualityPreference
Algorithm extension to selectQuality and the switchXQuality behaviors
Customer-policy config surface (qualityPreference)

Out of scope (separate features):

rendition-selection-caps — constraint+filter, different mechanism. Caps narrow the candidate set; this feature biases selection within it. Composable: caps run first, then signals bias.
[bandwidth-estimation] (candidate) — the dual-EWMA primitive that produces bandwidthState. Not changed by this feature; consumed unchanged.
[non-zero-pts-support] (candidate) and other cluster B candidates — time-domain concerns; orthogonal to signal-driven selection.

Out of scope (different architectural layer):

Customer-facing UI for quality-preference toggles (adapter-level). The SPF feature owns the state slot + algorithm; the adapter renders the UI ("Auto" / "Data saver" / "Best quality" menu items) and writes to the config.
Signal-availability detection at the platform level (Chrome on Mac vs Chrome on Android vs iOS Safari). The SPF feature handles graceful no-op when an API is unavailable; the adapter / consumer doesn't need to know about platform-specific availability.
Manual quality-pinning. That's the userVideoTrackSelection constraint+filter precedent from video-abr — orthogonal to multi- signal-ABR.

Likely cross-cutting impact

Things this feature probably forces decisions on, not just additions:

Signal-fusion algorithm shape. The current selectQuality is a pure function over candidates + bandwidth. Extending to multi-input requires a fusion strategy. Three shapes worth considering: (a) Bias-factor weighting — each signal produces a multiplier (e.g., effectiveBandwidth = measuredBandwidth × networkBias × batteryBias × thermalBias × preferenceBias); selectQuality runs against the effective bandwidth. Simple, composable, easy to disable per-signal by setting bias to 1.0. (b) Decision-tree / staged-filter — each signal further narrows the candidate set or shifts the bandwidth thresholds. More expressive; harder to compose. (c) Scoring function — each candidate gets a multi-input score; highest score wins. Most flexible; most opaque to tune. Lean: (a) for simplicity; revisit if real-world tuning needs more expressiveness.
Bandwidth safety floor. Non-bandwidth biases must not override the bandwidth-safety floor — even with 'best-quality' customer preference, the engine can't pick a rendition above measuredBandwidth × safetyMargin (it'll rebuffer). Bandwidth supremacy is structural.
Per-signal availability fallback. Each monitor behavior gracefully no-ops when the underlying browser API is unavailable. The state slot stays undefined; the algorithm tolerates undefined by skipping that signal's bias (bias factor → 1.0). This is the well-precedented pattern from userVideoTrackSelection (default-undefined). No composition-variant complexity needed.
Audio-vs-video signal applicability. Some signals apply uniformly (battery, network, customer-preference); some are video-specific (thermal-pressure, visibility). The algorithm extension is signal- shape-agnostic; per-type relevance is in the bias-factor calculation per signal. Audio-ABR ignores thermal and visibility because its bias factors are always 1.0 for those signals.
Monitor-behavior composition shape. Each signal monitor is a middle-pattern behavior. They compose into ABR-enabled engine variants alongside the algorithm. Two shapes possible: (a) compose all signal monitors into ABR variants (always-on with slot-undefined fallback for unavailable APIs); (b) compose opt-in per signal (each is a separate behavior consumers add to their engine variant). Lean: (a) for default discoverability; (b) is achievable by simply removing the monitor and letting the slot stay undefined.
Customer preference vs automatic biases. Customer preference is the final layer in the bias-factor stack: it should be able to bias upward ('best-quality' overrides battery-low downward bias) but never override the bandwidth safety floor. This is the "policy reigns over automatic biases, physics reigns over policy" layering.
Cross-feature with audio-abr. When audio-ABR lands, it consumes the same signal slots as video-ABR. Signal monitors don't duplicate; signal slots are shared. The fusion algorithm is per-type because the bias-factor calculations differ (audio doesn't bias on thermal/visibility); the infrastructure is shared.

Open questions

Fusion algorithm strategy. Bias-factor weighting (lean), decision- tree, or scoring function. Choice affects how customer-policy expressivity composes; bias-factors are easiest to tune empirically.
Default bias strengths. Per-signal, what's the default scalar? Network 'slow-2g' → 0.3? Battery level < 0.2 → 0.7? These need empirical tuning, but a reasonable default set must ship.
Thermal-pressure detection heuristic. Dropped-frames threshold alone vs combined with long-task signals. Distinguishing decode- pressure (real thermal/CPU issue) from network-driven gaps (which look similar in dropped-frames) is non-trivial. Naive depth: threshold-only. Full depth: combine signals + hysteresis.
Customer-preference shape. Single enum ('auto' / 'data-saver' / 'best-quality') vs scalar bias-strength (qualityBias: -1.0 to 1.0) vs per-signal customer overrides (more granular control). Customer use cases drive this; lean enum for discoverability + a Tier 2 scalar tuning slot for advanced consumers.
Audio-ABR signal subset finalization. Some signals (battery, network) clearly apply to both; some (thermal, visibility) are video- specific. Distinguish at the algorithm-extension level (audio ignores some inputs) or at the signal-slot-population level (audio doesn't read those slots). Implementation question.
Picture-in-Picture distinction. PiP makes document.visibilityState unreliable for "is the user watching?" — PiP is visible to user even when document is hidden. Visibility signal needs to fold in PiP state. Open: which APIs reliably surface PiP state across browsers, and what's the fallback when not detectable.
Signal-staleness handling. A monitor that reports effectiveType: '4g' once at startup and never updates is stale on a long playback session. Should the algorithm time-bound signal freshness, or trust monitors to keep slots updated? Lean: trust monitors (event-driven update is the precedent).
Cross-feature ordering with rendition-selection-caps. When both features ship, the pipeline is capability → caps → bandwidth-and- signals → selection. Filter order is implicit but worth confirming.

video-abr (prerequisite) — base algorithm being extended. selectQuality is the read-side; this feature widens its input axis.
audio-abr (parallel sibling) — extension applies to both video and audio ABR. Shared signal-monitor infrastructure; per-type bias calculation.
rendition-selection-caps — complementary feature on the constraint+filter axis. Caps narrow the candidate set; signals bias selection within it. Composable in the pipeline: caps → bandwidth+signals → selection.
buffer-management — when visibility signal lands, buffer-management may also want to pause forward-buffer fetching when hidden. Cross-cutting concern; lives in buffer-management not here.
[bandwidth-estimation] (candidate) — the dual-EWMA primitive for bandwidthState. Not changed by this feature; the existing bandwidthState slot is the bandwidth-side input to fusion.
[capability-probing] — adjacent for advanced cases (e.g., hardware-DRM security-level pressure may interact with multi-signal bias if a thermally-pressured device can't sustain hardware decode); out of scope for v1.

16 KiB Raw Blame History Unescape Escape