zoriya/v10
1
0
Fork 0
mirror of https://github.com/zoriya/v10.git synced 2026-08-16 02:45:09 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(spf): track switching design (#1631)

Browse Source 
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Christian Pillsbury
2026-06-08 14:39:40 -07:00
committed by GitHub
co-authored by Claude Opus 4.8
parent 057f32573e
commit f15d2b5e78
1 changed files with 410 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
internal/design/spf/track-switching-model.md
+410
View File
@@ -0,0 +1,410 @@
---
status: draft
date: 2026-06-04
---
# Track switching: the selection rule model
## The problem
Track selection started simple — one behavior writing each state signal, one default-picker per track
type. As features accumulate, many independent decision domains converge on the same selection signal
(`selectedVideoTrackId` / `selectedAudioTrackId`):
- Bandwidth-driven ABR
- Consumer-driven preference (an explicit user selection)
- Capability physics (a codec/DRM the environment can't decode)
- Codec policy (prefer/force AVC, surround vs stereo)
- Delivery policy (resolution caps, multi-signal ABR)
- CDN routing (failover, content steering)
- Default-language picking
Each new feature would otherwise need either another behavior competing to write the signal or a new
branch in a central picker — both lead to a monolith that grows a branch per feature. The substrate that
avoids this is a **pipeline**: a candidate track list flows through feature-contributed *rules* into a
composer that produces the ordered list whose first entry becomes the selection.
This document recommends a specific rule model for that pipeline — the shape of an individual *rule* and
how rules combine into a final pick. The two are a package: a rule shape implies a way of combining
rules. The model here is **a hard pruning pass followed by a chain of soft, narrowing rules and one
ranker** — Filter + Sort, with the hard exclusions split out into their own *constraints* phase and the
remaining rules allowed to narrow softly rather than only reorder.
```
candidate tracks
→ [ constraints ] hard pruning — anything that can't play here is gone
→ [ rule ] → [ rule ] … soft narrowing + ranking, in one ordered chain
→ pick the first surviving track
```
## The shape in two phases
**Constraints run first.** A constraint is rule-shaped, but its exclusion is *hard*: never attempt to
play a track it removes. Constraints are the things that mean "can't play here" — a codec or DRM the
environment can't decode, or a CDN that's failing and in cooldown. They are pooled and applied up front,
before any preference is consulted. Because every constraint only ever removes the unplayable, the order
they run in can't change the outcome, and a track that survives the pass is guaranteed playable.
**Rules run next, in one ordered chain.** A rule over the surviving set is one of two things:
- A **soft filter** narrows the running set to the tracks it wants — *unless that would empty the set, in
which case it's skipped and the set is left as it was* (it falls through). Two soft filters compose by
narrowing one after the other, each against what the previous one left. A soft filter is how a
*preference* is expressed: restrict to what's wanted, but never to nothing.
- A **sort** reorders the running set by some criterion. Sorts here are *stable* — a stable sort leaves
the existing order untouched among tracks it can't tell apart, so an earlier sort's ordering survives as
the tiebreak under a later one. `abr` is the ranker: of the fitting tracks, highest quality first;
over-throughput tracks after, least-over first.
The pick is the first track left in the running set after the whole chain has run.
**Early-bail.** A soft filter that narrows the set to exactly one track has already decided the pick —
there's nothing left to choose among, so the chain can stop there. This is the same rule as "skip a
filter that would empty the set," read from the other end: zero survivors means skip, one survivor means
done. Placing the `abr` ranker last means narrowing finishes first, so a single-survivor bail happens
*before* the expensive throughput estimate ever runs.
### Which end of the chain is authoritative
In a chain of pure sorts, the *last* sort wins — it gets the final say on order. Soft filters invert
that. When two soft filters disagree — say a user picked 1080p and a codec policy prefers AVC, and no
track is both — the one applied **first** wins: it narrows to 1080p, then the AVC filter finds nothing
matching within that set and falls through, leaving the user's 1080p intact. Applied in the other order,
AVC would win and the user's pick would be the one to fall through.
So the rule chain reads **most authoritative first**: the user's explicit selection ahead of policy,
policy ahead of defaults, with the `abr` ranker last (its position doesn't fight anything — a sort only
reorders what survives, and putting it last is what enables early-bail). This is the reverse of the
last-sort-wins reading, and it's a direct consequence of preferences narrowing rather than reordering.
Read "most authoritative first" as a **rough heuristic for ordering the chain, not a fixed contract.**
It's generally right, and a good default to reach for, but the exact order is expected to shift with real
use cases and learnings — a given user selection might sensibly sit *after* some policy rule, and rules
that interact with `abr`'s fits-vs-over split (a resolution cap, for one) complicate the simple
authority gradient. The order is a starting assumption to refine, not a settled ranking.
## The rule set we're designing for
To judge the model on more than the toy example, here is the working set of selection concerns we
*anticipate* features contributing, drawn from feature planning across the project.
**Treat it as illustrative, not a specification.** Most of these aren't implemented yet; several aren't
precisely defined; and some will likely become more than one rule once they are — `rendition-selection-caps`,
for one, is really an upper cap, a hard floor, and a soft floor, which behave differently (see
[the known weakness](#the-known-weakness-ranked-fallback)). The "kind" assigned to each is a
best-current-guess. The point isn't the exact roster — it's that the set is *varied and growing*, which
is what stresses the model.
Each concern is sketched as one of three kinds — a **constraint** (hard exclusion of the unplayable), a
**scope** (a soft filter: a preference we'd narrow to, and rebuffer to honor), or a **ranker** (a sort by
a runtime signal):
| Feature (anticipated) | Likely kind | Roughly what it wants |
|---|---|---|
| `capability-probing` | constraint | Exclude tracks the environment can't decode (codec/DRM). |
| `multi-cdn-failover` | constraint + scope | Exclude a failed CDN's tracks during cooldown; prefer the active CDN's tracks. |
| `content-steering` | scope | Prefer the active pathway's tracks. |
| `video-abr` / `audio-abr` | ranker | Of the fitting tracks, highest quality; over-throughput as fallback. |
| `multi-signal-abr` | ranker | Order by a fused quality/throughput signal. |
| `userVideoTrackSelection` / `userAudioTrackSelection` | scope | Honor an explicit user choice above all policy — usually of a track *feature* (a video resolution, an audio language), so it prefers every track matching that feature, not one specific track. (The names imply a single-track pick; that's an artifact of today's naive implementation.) |
| `multi-language-audio` | scope | Prefer the default/selected language. |
| `5.1-surround-selection` | scope | Prefer the requested channel layout. |
| `hevc-variant-selection` | scope | Prefer (or force) a codec, e.g. AVC over HEVC. |
| `rendition-selection-caps` | scope (likely several) | Stay within an upper cap (player/screen size, cost tier) and/or above a floor. |
This set is already long, and it will grow as features land. That breadth — not any single entry — is
the backdrop for the model.
## A shared example
A compact subset of those concerns, on one track set, used to watch the model work.
```ts
const tracks = [
// Spanish — E-AC-3 (surround, runs hotter than AAC)
{ id: 'esLowEac', lang: 'es', codec: 'eac3', bitrate: 192 },
{ id: 'esHighEac', lang: 'es', codec: 'eac3', bitrate: 384 },
{ id: 'esMaxEac', lang: 'es', codec: 'eac3', bitrate: 640 },
// English — AAC
{ id: 'enLowAac', lang: 'en', codec: 'aac', bitrate: 64 },
{ id: 'enHighAac', lang: 'en', codec: 'aac', bitrate: 256 },
{ id: 'enMaxAac', lang: 'en', codec: 'aac', bitrate: 512 },
];
```
| # | Rule | Kind | The concern |
|---|------|------|-------------|
| 1 | `preferredLanguage` (a `multi-language-audio` / user pick) | scope | Want every track in the chosen language; no opinion on bitrate; other languages acceptable but not wanted. |
| 2 | `capabilityProbing` | constraint | Exclude unplayable tracks; no preference among the playable rest. |
| 3 | `abr` | ranker | Of the fitting tracks, highest quality; over-throughput tracks a fallback, least-over first. |
A pick depends on the playback **context** — one fact per rule:
- **Preferred language** — Spanish.
- **E-AC-3 decodable?** — yes.
- **Throughput estimate** — ~300 kbps, so `esLowEac` / `enLowAac` / `enHighAac` fit; the rest are over.
The correct pick is **`esLowEac`** — the user wants Spanish, and `esLowEac` is the only Spanish
rendition that fits. Note the trap: the best *fitting* track overall is `enHighAac` (256 > 192), so any
model that lets throughput quietly outrank language picks English here — wrong.
## How the model works on the example
**Constraints first.** `capabilityProbing` is the only constraint, and E-AC-3 decodes, so it removes
nothing. All six tracks survive to the rule chain.
**Then the chain, most authoritative first** — `preferredLanguage` (scope) ahead of `abr` (ranker, last):
```ts
let results = applyConstraints(tracks); // → all six (everything is playable)
results = preferredLanguage(results); // soft filter → Spanish
// → [esLowEac, esHighEac, esMaxEac] English dropped; Spanish is non-empty so the narrow sticks
// (3 left, not 1 → no early bail; keep going)
results = abr(results); // sort → fitting first, then least-over
// → [esLowEac, esHighEac, esMaxEac] esLowEac (192, fits) ahead of the over-throughput two
pick = results[0] = esLowEac // ✅ language narrowed the field; abr ordered within it
```
`preferredLanguage` removes English outright, so `abr` never gets the chance to float `enHighAac` to the
front — it only orders the Spanish tracks that survived. And the order of the two rules doesn't matter:
run `abr` first and it sorts all six, then `preferredLanguage` narrows to the Spanish three *keeping
abr's order among them* — same pick. A preference that **narrows** can't be overridden by a ranker the
way a preference expressed as a sort could be, which is the whole reason scopes are soft filters here and
not sorts.
## Both track types: the two chains
The shared example is small. The model has to carry the broader set above. Here is each track type's
arrangement, with the same illustrative-not-final caveat as the table — and the same caveat on chain
order from above: the most-authoritative-first sequence shown is a working starting point, not a fixed
ranking.
```
AUDIO → selectedAudioTrackId
CONSTRAINTS (hard, pooled, order-free):
capability-probing (codec/DRM) · multi-cdn-failover (failed CDN, cooldown)
RULE CHAIN (most authoritative → least; abr last):
user userAudioTrackSelection chosen language ← most authoritative
scope multi-language-audio default language
scope 5.1-surround requested channel layout (rebuffer to honor)
scope content-steering active pathway (reflects an upstream pick) [shared]
scope multi-cdn-failover active CDN (reflects an upstream pick) [shared]
ranker audio-abr fitting > over-throughput
```
```
VIDEO → selectedVideoTrackId
CONSTRAINTS (hard, pooled, order-free):
capability-probing (codec/DRM) · multi-cdn-failover (failed CDN) · (future) dropped-frames
RULE CHAIN (most authoritative → least; abr last):
user userVideoTrackSelection chosen resolution ← most authoritative
scope hevc / force-AVC preferred codec (rebuffer to honor)
scope rendition cap (upper) within cap (lower tracks fit → never rebuffers)
scope rendition floor (hard) ≥floor (rebuffer rather than show too-low)
scope content-steering active pathway (reflects an upstream pick) [shared]
scope multi-cdn-failover active CDN (reflects an upstream pick) [shared]
ranker video-abr / multi-signal-abr / (future) dropped-frames
── not placeable ──
✗ rendition floor (soft) "prefer ≥Y but don't rebuffer" → ranked fallback (see below)
```
Two notes the chains depend on:
- **Pathway rules reflect an upstream pick.** A separate, session-level behavior owns *which* CDN/pathway
is active (it identifies pathways, picks one, fails over). `content-steering` / `multi-cdn-failover`
don't choose — they *reflect* that state: prefer the active pathway (a scope) and exclude a failed one
(a constraint). Throughput shifts from a switch are read by `abr` downstream; the two never form a
loop.
- **Cross-type consistency is a composition convention.** The audio and video rule sets include the
*same* pathway rule definition, so both reflect the same upstream pick and stay on one pathway.
Mis-wiring a different pathway rule per type is the only failure mode — a convention tolerates it.
Why these land cleanly as soft filters:
- An **upper cap** narrows to the within-cap tracks; the lower tracks always fit, so the pick never
rebuffers — the cap runs *with* `abr`'s ceiling.
- A **hard floor** narrows to `≥floor`; if every `≥floor` track is over-throughput, `abr` picks the
least-over and the player rebuffers — which is what a hard floor asks for.
- A **scope that rebuffers to honor** (preferred language, channel layout, codec) narrows to the wanted
tracks; if all of them are over-throughput, you rebuffer to stay in scope. Correct, and it can't be
mis-ordered into picking an over-throughput track it *didn't* want, because it removes rather than
reorders.
The single row that doesn't fit is the `✗` line.
## The known weakness: ranked fallback
There is one shape this model can't express. Call it **ranked fallback**: a preference that has a first
choice *and* an ordered second choice for when the first choice isn't available — and whose second choice
still has to respect which tracks actually fit the current throughput. The model can *narrow* (keep or
drop a track) and it can *rank* (one ordering of what's left), but it has no way to say "prefer these,
ordered this way; failing that, fall back to those, ordered that way."
The clearest case is a **soft minimum-resolution floor**: "prefer ≥360p, but don't rebuffer for it." On a
240/360/480 ladder, the desired picks across falling throughput are `360, 360, 240` — take 360p while it
fits, drop to 240p rather than rebuffer once it doesn't.
- As a **soft filter** (`≥360p`): when only 240p fits, the `≥360p` tracks are all over-throughput, so
`abr` picks an over-throughput one and the player rebuffers — exactly what we said not to do.
- As a **sort** placed before `abr`: it can float 360p ahead, but it can't *lift* 240p to 360p when only
240p fits, so it does nothing useful in the case that matters.
The floor needs to sit *between* `abr`'s fits-vs-over decision and its bitrate ordering — rank the
fitting tracks one way, fall through to a *ranked* leftover another way — and neither a soft filter (keep
or drop) nor a single stable sort can straddle that. A verdict/tier-based model, where a rule can mark a
ranked preferred set *and* a ranked fallback set, expresses it natively; this model trades that
expressiveness for two simple rule kinds.
**We are choosing to bite this bullet.** A soft resolution floor is an ordinary product ask, not a
corner case, so this is a real (bounded) gap, not a hypothetical one. We are not solving it here — it
stands as a known limitation, and if it becomes load-bearing the escape hatch is the same one every
engine that fuses tier and ranking uses: split `abr` into a bitrate ordering and a fits-vs-over
threshold, and slot the floor between them by hand. That's a rewrite of an existing rule, which is why
it's a weakness and not a free extension — but it's a localized one, and it buys back the two-primitive
simplicity everywhere else.
## Why this model
The recommendation is this Filter + Sort variation — constraints as a hard pre-pass, preferences as soft
filters, one ranker, one ordered chain — over a richer per-track verdict model, on these grounds:
- **Two rule kinds, plainly composed.** A constraint excludes the unplayable; a rule narrows or ranks.
There is no verdict algebra to define or get right; the composer drops the constraints' exclusions,
then runs the chain.
- **Preferences narrow, so order is mostly free.** Because a scope removes rather than competes in a sort
key, it can't be mis-ordered into an unsafe or wrong pick against the ranker — the language-vs-abr trap
that a pure sort chain has simply doesn't arise. Order matters only among soft filters that genuinely
conflict (resolved most-authoritative-first) and between two rankers that disagree.
- **Safety is structural.** Playability is decided once, up front, by the constraints pass, independent
of any preference or its ordering. A surviving track is playable by construction.
- **Early-bail is natural.** Narrowing to one track ends the chain before the throughput estimate runs.
- **The cost is one named gap** — ranked fallback (the soft floor) — accepted with eyes open, with a
localized escape hatch if it ever has to be closed.
The trade is deliberate. The main alternative considered was a richer model in which each rule emits a
per-track verdict (a track is preferred, merely allowed, or excluded) and a composer combines those
verdicts by category. That model expresses ranked fallback natively and shaves the residual
order-sensitivity, so it would make *some* rule compositions solvable that this one can't — the soft
floor among them. But it's a narrow set: it doesn't unlock *most* compositions (the two models reach the
same pick on essentially every rule we anticipate), and it pays for that headroom with a more
complicated rule shape *and* a more complicated composition engine to combine the verdicts. For the rule
set we anticipate, the simpler two-kind model covers everything but the one shape above, covers it more
robustly under author error, and keeps both the rules and the engine that combines them small. Status
stays `draft` pending that call.
## Fitting the model to the track-switching behavior
This is a sketch of how the model lands in the running engine, grounded in the track-switching behavior
as it stands today — *not* a settled wiring. Like the chain-order heuristic above, expect it to firm up
as the rule set does.
The behavior already runs a miniature of this model. While a presentation is resolved, it reacts: it
takes the tracks of its type, narrows them by a single user-selection preference (falling back to all of
them if that preference matches nothing), and — if the narrowing leaves exactly one track — sets that as
the pick and stops, *without ever reading the bandwidth estimate*; otherwise it runs the bandwidth-driven
ranker. That is one soft filter, the early-bail, and one ranker, hardcoded. The model generalizes it in
two places: it adds the constraints pass, and it replaces the single fixed filter-then-rank with the
open-ended chain. Rules, like the behavior's existing picker and tuning, arrive by configuration — a
feature contributes its rule without reopening the behavior.
Three things the model implies about where each piece lives:
- **Constraints gate; rules run in the reaction.** Constraints are a derived value computed *outside* the
reaction — they answer "what's playable here at all," the same kind of question that already decides
whether the behavior is in its selecting state. So they feed the gate: while the playable set is still
unknown the behavior isn't ready to select; once it's known, the behavior enters its selecting state and
hands the pruned set in as the candidate list. The rules, by contrast, run *inside* the reaction, over
those candidates.
- **An empty playable set is its own outcome.** A non-empty playable set is the candidate list; the
still-unknown case gates as above. The case worth calling out — and still open — is when constraints
leave *nothing* playable. That is not a fall-back-to-everything (that's the *soft* filter's job;
constraints are hard): "nothing decodable here" is a terminal condition the engine surfaces, a distinct
not-ready state rather than a pick. Exactly how that state is modeled is open.
- **Rules are reactive, and early-bail prunes what they react to.** Almost every rule reads more than the
track list — a ranker reads the throughput estimate, a pathway scope reads which CDN is active. Those
reads happen inside the reaction, so the reaction subscribes to exactly the signals the rules consulted
and re-runs when any of them change. This is why early-bail is more than skipping the throughput
estimate once: a rule that narrows the field to a single track ends the chain *before* later rules read
their signals at all, so the reaction never subscribes to them and never re-fires on their changes while
that narrowing holds. The behavior already does exactly this by hand for the bandwidth estimate; the
chain makes it the general rule.
- **How a rule expresses its signal dependencies is an open contract question.** Because a rule reads more
than the track list, it has to surface *which* state and context signals it consults so the reaction can
subscribe to them — and because the behavior declares its dependencies up front while rules arrive by
configuration, how a rule's dependencies reach that declaration isn't settled. Some coarse options, not
yet chosen:
- **Declared, like a behavior** — a rule is an object (or a type) that names the signals it requires, or
might use, the way a behavior declares its own; composition aggregates those into the behavior's
declared set. Most explicit, composes cleanly, most up-front machinery.
- **Tightly coupled reads** *(the likely starting point)* — the rule handed in via configuration is a
small closure that reads its signals directly, and the behavior's declared set is just what those rules
read. The selection logic itself can still live decoupled in the media layer as a plain function tied
to no signals; only the thin rule wrapper that does the reading is coupled. Simplest to start; defers
the general contract.
- **A prebuilt selection context** — the behavior reads a fixed surface of signals once per run and hands
each rule a plain context value, so rules stay pure and never touch signals. Keeps the behavior's
declaration static, but bounds a rule to whatever that fixed surface carries.
This also interacts with early-bail, which wants per-run lazy reads in chain order rather than an
up-front union of every rule's signals. Out of scope here; noted because it likely shapes the eventual
rule contract.
## Appendix A: prior art
Every major HTTP-adaptive-streaming engine separates **hard constraints (a pruning pass) from soft
ranking**, and none combines decision sources by weighted-sum scoring. That universal split is what
grounds the two-phase shape here — constraints then rules is the form the whole field converged on.
| Engine | Hard exclusion | Soft combination policy | Criteria are… |
|---|---|---|---|
| [**Shaka**](https://github.com/shaka-project/shaka-player) | `getPlayableVariants()` gate (`filterManifest` + `applyRestrictions`); two typed errors | preference narrows the set, then a *single* bandwidth ranker (`SimpleAbrManager`) | a fixed pipeline |
| [**hls.js**](https://github.com/video-dev/hls.js) | codec-unsupported levels removed from the manifest | one ABR ranker + min/max **index clamping** (cap-level, HDCP, min-bitrate each write a clamp) | hardcoded controllers |
| [**VHS**](https://github.com/videojs/http-streaming) | `excludeUntil === Infinity` (`isIncompatible` / `isEnabled`) | bandwidth + resolution sort with a fallback chain | one selector function |
| [**dash.js**](https://github.com/Dash-Industry-Forum/dash.js) | `_filterByPossibleBitrate` / `_filterByPortalSize`; `CapabilitiesFilter` | rules collection → `getMinSwitchRequest`: most-cautious pick within a priority tier (STRONG / DEFAULT / WEAK) | a fixed rule list |
| [**rx-player**](https://github.com/canalplus/rx-player) | `isRepresentationPlayable` (decipherable + codec); representation locking narrows to an ID set | soft throttle / resolution limits (with fallback), then a fixed precedence cascade of bandwidth / buffer / guess estimators | a hardcoded selector |
| [**OSMF**](https://sourceforge.net/adobe/osmf/svn/HEAD/tree/) | `maxAllowedIndex` applied after the rules | `SwitchingRules` → manager takes the most conservative recommended index (`-1` = no opinion) | a fixed rule list |
| [**media3 / ExoPlayer**](https://github.com/androidx/media) | `SELECTION_ELIGIBILITY_NO` (capability, or constraints when relax is off) | multi-key comparison (`ComparisonChain`) — each key breaks ties left by the one before; soft constraints fall back gracefully (`exceedConstraintsIfNecessary`) | a hardcoded comparator |
| [**VLC**](https://github.com/videolan/vlc) | `RepresentationSelector` resolution / bitrate bounds (shared filter pass, fallback-to-lowest) | a single `AbstractAdaptationLogic` chosen up front by config (`adaptive-logic`) — not composed | a swappable whole-algorithm strategy |
Four things this tells us:
1. **The hard/soft split is universal**, not our invention — it *is* the constraints-then-rules shape
here. A hard exclusion before any ranking is the declarative form of what every engine encodes
imperatively as "survived the pruning pass."
2. **No one adds up the criteria into a single score.** dash.js and OSMF take the most cautious
recommendation across rules within a priority tier; ExoPlayer ranks on one criterion and uses the next
only to break ties; rx-player consults its estimators in a fixed order of precedence. A
"give every track a number and add the criteria up" approach has no precedent and would mix things
that can't share one scale — a language preference and a throughput estimate aren't measured in the
same units.
3. **This model's shape is battle-tested.** A pruning pass plus a chain that narrows then ranks, with a
soft preference that falls back gracefully when it would leave nothing, is essentially ExoPlayer's
`ComparisonChain` with `exceedConstraintsIfNecessary`, rx-player's narrowing-with-fallback cascade, and
Shaka's filter-then-rank. The ranked-fallback shape we're giving up is the one dash.js / OSMF get from
their priority tiers — which is the alternative we're deliberately not adopting.
4. **What none of them have is per-rule composability.** Every engine's combination logic is a *central,
hardcoded* function — ExoPlayer's comparator, dash.js's `getMinSwitchRequest`, OSMF's `checkRules`,
rx-player's `getCurrentEstimate`, hls.js's clamps. VLC is the only pluggable one, but at the
whole-algorithm level, which can't express a concern that must *stack* with ABR rather than replace it
(multi-CDN failover isn't "a different ABR algorithm"). That's fine for a player with a fixed feature
set; SPF is a composition framework where features arrive *by composition*, so the combination policy
has to be expressible as independent, feature-owned rules. The engines establish that the underlying
policy is sound; the per-rule composition is what this model adds on top.
*Scope: surveyed independent JS engines, native mobile (media3), native desktop (VLC), and the Flash-era
ancestor (OSMF). Excluded UI wrappers (Vidstack, Plyr, Media Chrome, Mux Player) — they delegate
selection to one of these — and native black boxes (AVPlayer / Apple HLS) that aren't inspectable.*