mirror of
https://github.com/zoriya/v10.git
synced 2026-08-16 02:45:09 +00:00
feat(spf): multi cdn failover (#1671)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Christian Pillsbury
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
33873bc4ae
commit
b89f1e944c
@@ -1,7 +1,7 @@
|
||||
---
|
||||
status: draft
|
||||
date: 2026-06-05
|
||||
definition: sketched
|
||||
date: 2026-06-08
|
||||
definition: implemented
|
||||
---
|
||||
|
||||
# Multi-CDN failover
|
||||
@@ -22,12 +22,16 @@ Two sub-features, mapping onto the two rule kinds in that model:
|
||||
so video / audio / text resolve from one host. This is the
|
||||
*active-pathway scope* the track-switching model lists for
|
||||
multi-cdn-failover.
|
||||
2. **Failover** *(deferred)* — when requests to the active CDN fail too
|
||||
often within a window, a **constraint** removes that CDN's tracks from
|
||||
the candidate set during cooldown and the session behavior rotates the
|
||||
active CDN. This is the *failed-CDN constraint* the model lists, and it
|
||||
consumes [network-resilience](./network-resilience.md)'s per-host
|
||||
circuit-breaker — its hard prerequisite.
|
||||
2. **Failover** *(implemented)* — **site-adds, behavior-expires.** Fetch
|
||||
sites add a CDN to a `failedCdns` set on a failed fetch (the *trip*); a
|
||||
shared **constraint** (`excludeFailedCdns`) prunes that CDN's tracks from
|
||||
the candidate set so the scope falls to the next CDN; `setupFailoverMonitor`
|
||||
removes the CDN once a cooldown lapses (the *expiry*) and the scope returns
|
||||
to it. This is the *failed-CDN constraint* the model lists. It shipped
|
||||
**self-contained** — a cooldown timer, *not*
|
||||
[network-resilience](./network-resilience.md)'s circuit-breaker. Retries are
|
||||
a future refinement that would sit *below* the trip (so it sees post-retry
|
||||
terminal failures), not a prerequisite.
|
||||
|
||||
A **Media-src feature** for sources that genuinely require failover, with
|
||||
a **Player feature** surface at the failover tier (customer-customizable
|
||||
@@ -36,26 +40,30 @@ ambiguity reflects that split.
|
||||
|
||||
## Status
|
||||
|
||||
- **Composition:** sub-feature 1 (sticky CDN pick) is implemented in
|
||||
`createSimpleHlsEngine` and `createHlsAudioOnlyEngine`. The
|
||||
`resolveCdnPriority` behavior owns the `cdnPriority` signal (the
|
||||
manifest-ordered CDN list, most-preferred first); the `preferActiveCdn`
|
||||
scope rule (shared by the video + audio chains in `track-switching`)
|
||||
narrows candidates to the highest-priority CDN with surviving tracks.
|
||||
Failover (sub-feature 2) is not implemented — no constraints pass, no
|
||||
per-CDN failure tracking, no rotation.
|
||||
- **Definition depth:** sketched — sub-feature 1 has a populated
|
||||
implementation surface + verification; sub-feature 2 stays at the
|
||||
scope-and-constraints level pending its prerequisite.
|
||||
- **Hard prerequisite (failover only):**
|
||||
[network-resilience](./network-resilience.md). The failed-CDN
|
||||
constraint consumes the foundation's per-host circuit-breaker /
|
||||
retry-exhaustion state. Sub-feature 1 has no such dependency — it's
|
||||
pure selection over the already-parsed candidate set.
|
||||
- **Composition:** both sub-features are implemented in
|
||||
`createSimpleHlsEngine` and the audio-only engine. `deriveCdnPriority`
|
||||
owns the `cdnPriority` signal (manifest-ordered CDN list); the
|
||||
`preferActiveCdn` scope rule narrows candidates to the highest-priority CDN
|
||||
with surviving tracks (shared by the video + audio chains). For failover:
|
||||
the `excludeFailedCdns` constraint prunes tracks whose CDN is in the
|
||||
`failedCdns` set; fetch sites trip a CDN into `failedCdns` on a failed fetch
|
||||
(`failoverFetch` for media playlists in `resolve-track`, `failoverFetchBytes`
|
||||
for segments in `setup-buffer-actors`); `setupFailoverMonitor` owns `failedCdns`
|
||||
and removes each CDN once its cooldown lapses.
|
||||
- **Definition depth:** implemented.
|
||||
- **Detection (self-contained, no hard prerequisite):** a CDN trips on the
|
||||
**first terminal fetch failure** (network error or non-OK status); cooldown
|
||||
is the only back-off. `setupFailoverMonitor` is a per-source cooldown timer,
|
||||
*not* a circuit-breaker imported from
|
||||
[network-resilience](./network-resilience.md). Retries are a future
|
||||
refinement *below* the trip (so it would observe only post-retry terminal
|
||||
failures), not a dependency — this feature ships without
|
||||
`network-resilience` existing.
|
||||
- **Governing model:** [track-switching-model.md](../track-switching-model.md)
|
||||
— multi-CDN is the canonical *constraint + scope* feature there. The
|
||||
active-CDN scope is a soft filter in the rule chain; the failed-CDN
|
||||
constraint is a hard filter in the (not-yet-built) constraints pre-pass.
|
||||
— multi-CDN is the canonical *constraint + scope* feature there. Both halves
|
||||
are now built: the active-CDN scope (soft filter in the rule chain) and the
|
||||
failed-CDN constraint (hard filter in the constraints pre-pass —
|
||||
`applyConstraints` + the `constraints` config slot now exist).
|
||||
|
||||
## How redundant streams are modeled
|
||||
|
||||
@@ -65,7 +73,8 @@ redundant-streams manifest lists each rendition once per CDN (duplicate
|
||||
existing parser already emits one `Track` per entry with a unique id and
|
||||
its own absolute `url`. So the candidate set for each type *already*
|
||||
contains one variant per CDN. CDN identity is derived from each track
|
||||
URL's origin (`getCdnId`); the set of CDNs is published as a single
|
||||
URL's origin (`getCdnId`, overridable via engine config — e.g. to key on
|
||||
Mux's `cdn=` query param); the set of CDNs is published as a single
|
||||
per-presentation ordered signal (`cdnPriority`, most-preferred first —
|
||||
mirroring HLS content steering's `PATHWAY-PRIORITY`). The *active* CDN is
|
||||
not stored: the scope derives it as the highest-priority `cdnPriority`
|
||||
@@ -74,38 +83,39 @@ CDN-tagged track id means `resolveTrack` / segment loading fetch from that
|
||||
CDN with no further plumbing.
|
||||
|
||||
This list shape is what makes failover fall out cleanly: the failed-CDN
|
||||
constraint (sub-feature 2) prunes a cooled-down CDN's tracks, so
|
||||
"first-with-survivors" moves to the next CDN automatically and returns to
|
||||
the primary when it recovers — no reactive rewrite of an "active" value.
|
||||
Content steering, likewise, just reorders `cdnPriority` (pathway priority
|
||||
as a sort key).
|
||||
constraint prunes a cooled-down CDN's tracks, so "first-with-survivors"
|
||||
moves to the next CDN automatically and returns to the primary when it
|
||||
recovers — no reactive rewrite of an "active" value. Content steering,
|
||||
likewise, just reorders `cdnPriority` (pathway priority as a sort key).
|
||||
|
||||
This is why sub-feature 1 needed **no parser change** and no new data
|
||||
shape — only a selection behavior and a scope rule over existing tracks.
|
||||
This is why the feature needed **no parser change** and no new data shape
|
||||
— only selection behaviors, a scope rule, and a constraint over existing
|
||||
tracks, plus a fetch decorator that records failures into `failedCdns`.
|
||||
|
||||
## Phases of complexity
|
||||
|
||||
| Phase | Sub-feature | Kind | What | State |
|
||||
|---|---|---|---|---|
|
||||
| Sticky CDN pick | 1 | scope | `resolveCdnPriority` publishes the manifest-ordered CDN list (`cdnPriority`); `preferActiveCdn` narrows every type's candidates to the highest-priority CDN with surviving tracks, falling through when nothing matches. Shared list → all types on one CDN | **Implemented** |
|
||||
| Failed-CDN constraint | 2 | constraint | A constraint in the track-switching constraints pre-pass removes a cooled-down CDN's tracks from the candidate set. The scope then picks the next `cdnPriority` entry automatically. Requires the generic constraints phase (track-switching-model "Phase 2") to be built first | Deferred |
|
||||
| Per-CDN failure tracking | 2 | — | Count per-CDN fetch failures within a window; mark a CDN unhealthy / in cooldown. Consumes `network-resilience`'s circuit-breaker | Deferred (prereq) |
|
||||
| CDN priority override / steering | 2 | scope | Reorder `cdnPriority` to bias the pick (region-preferred, weighted, or content-steering's pathway priority). No reactive "active" rewrite needed — the order *is* the policy | Deferred |
|
||||
| Customer CDN-id derivation | 2 | config | Pluggable `getCdnId` for non-origin identity. The origin-based default is built in; a config seam is anticipated | Deferred |
|
||||
| Sticky CDN pick | 1 | scope | `deriveCdnPriority` publishes the manifest-ordered CDN list (`cdnPriority`); `preferActiveCdn` narrows every type's candidates to the highest-priority CDN with surviving tracks, falling through when nothing matches. Shared list → all types on one CDN | **Implemented** |
|
||||
| Constraints pre-pass | 2 | — | `applyConstraints` + the `constraints` config slot in `setupTrackSwitching` (the hard-filter pre-pass that runs before the rule chain). Reusable by capability-probing | **Implemented** |
|
||||
| Failed-CDN constraint | 2 | constraint | `excludeFailedCdns` prunes tracks whose CDN ∈ `failedCdns`; the scope falls to the next `cdnPriority` entry and snaps back on recovery | **Implemented** |
|
||||
| Per-CDN failure tracking | 2 | — | **Site-adds, behavior-expires**: fetch sites trip a CDN into `failedCdns` on a failed fetch (`failoverFetch` / `failoverFetchBytes`); `setupFailoverMonitor` expires it after a cooldown. Self-contained (trip-on-first-failure + cooldown), not a `network-resilience` circuit-breaker | **Implemented** |
|
||||
| Customer CDN-id derivation | 2 | config | Pluggable `getCdnId` (engine config) for non-origin identity, threaded to all four CDN-id sites; origin-based default | **Implemented** |
|
||||
| CDN priority override / steering | 2 | scope | Reorder `cdnPriority` to bias the pick (region-preferred, weighted, or content-steering's pathway priority). No reactive "active" rewrite needed — the order *is* the policy | Deferred (content-steering) |
|
||||
|
||||
## What's in scope vs out of scope
|
||||
|
||||
**In scope:**
|
||||
- Sticky per-presentation CDN selection (sub-feature 1, done)
|
||||
- Failover via a track-switching constraint + active-CDN rotation
|
||||
(sub-feature 2)
|
||||
- Per-CDN health derived from `network-resilience`'s circuit-breaker
|
||||
- Customer-configurable CDN-id derivation / rotation policy
|
||||
**In scope (all implemented):**
|
||||
- Sticky per-presentation CDN selection
|
||||
- Failover via the failed-CDN constraint + derived active-CDN rotation
|
||||
- Self-contained per-CDN failure tracking (trip-on-first-failure + cooldown)
|
||||
- Customer-configurable CDN-id derivation (`getCdnId`)
|
||||
|
||||
**Out of scope (separate cluster G sister features):**
|
||||
- **[network-resilience](./network-resilience.md)** *(foundation,
|
||||
prerequisite for failover)* — retry + backoff + circuit-breaker.
|
||||
Multi-CDN consumes; doesn't reimplement.
|
||||
- **[network-resilience](./network-resilience.md)** *(optional future
|
||||
refinement, not a prerequisite)* — retry + backoff + error-classification.
|
||||
Would sit *below* the failover trip to reduce false trips; failover ships
|
||||
without it.
|
||||
- **[content-steering](./content-steering.md)** — HLS content-steering
|
||||
protocol (server-advertised, dynamically-updated host pool). Different
|
||||
mechanism than static redundant streams, but the *same* active-pathway
|
||||
@@ -129,7 +139,7 @@ shape — only a selection behavior and a scope rule over existing tracks.
|
||||
video and audio chains reference the *same* `preferActiveCdn` definition
|
||||
reading the *same* list, so they agree on the CDN even if their per-type
|
||||
track arrays differ. (The doc's earlier per-rendition lean is superseded.)
|
||||
- **`cdnPriority` writer composition.** `resolveCdnPriority` is the sole
|
||||
- **`cdnPriority` writer composition.** `deriveCdnPriority` is the sole
|
||||
writer today (publishes the manifest order). Failover needs no second
|
||||
writer — the failed-CDN constraint prunes tracks and the scope re-derives
|
||||
the active CDN. Content-steering would *reorder* `cdnPriority` (still a
|
||||
@@ -140,120 +150,153 @@ shape — only a selection behavior and a scope rule over existing tracks.
|
||||
the primary. This is why the array beats a single reactive `activeCdn`
|
||||
value — the failed-set information is applied once (in the constraint),
|
||||
not duplicated into an active-value rewrite.
|
||||
- **Constraints phase is a shared prerequisite.** The failed-CDN
|
||||
constraint can't land until the generic constraints pre-pass
|
||||
(`applyConstraints` + the `constraints` config field + empty-playable-set
|
||||
terminal state) is built into `setupTrackSwitching`. The seam exists
|
||||
(`candidateSet` computed); the machinery does not. capability-probing
|
||||
shares this prerequisite.
|
||||
- **Constraints pre-pass (now built).** The failed-CDN constraint runs in
|
||||
`setupTrackSwitching`'s `applyConstraints` pre-pass — the `constraints`
|
||||
config slot, applied to `candidateSet` before the rule chain.
|
||||
capability-probing can reuse it. One piece is deliberately *not* built: a
|
||||
terminal "everything pruned" state — today an all-CDNs-failed candidate set
|
||||
is empty and the prior pick is left in place (see *Follow-up candidates*).
|
||||
- **Live + multi-CDN.** During live playback the reload loop re-resolves
|
||||
the presentation; `resolveCdnPriority` re-publishes only when the CDN set
|
||||
the presentation; `deriveCdnPriority` re-publishes only when the CDN set
|
||||
changes (idempotent for a stable manifest). Cross-feature with
|
||||
[live-stream-support](./live-stream-support.md) (not yet implemented).
|
||||
- **Priority state across source changes.** `cdnPriority` tears down with
|
||||
the source via the resolved/unresolved cascade (per
|
||||
[source-replacement](./source-replacement.md)); per-host circuit-breaker
|
||||
state in `network-resilience` may outlive a source.
|
||||
- **Failover state is per-source.** Both `cdnPriority` and `failedCdns` tear
|
||||
down with the source via the resolved/unresolved cascade (per
|
||||
[source-replacement](./source-replacement.md)) — `setupFailoverMonitor` clears
|
||||
`failedCdns` and its cooldown timers on unload, so no failover state leaks
|
||||
across sources. (If `network-resilience` lands later, its per-host state may
|
||||
choose to outlive a source — that's its call, not failover's.)
|
||||
|
||||
## Open questions
|
||||
|
||||
- **Constraints-phase shape.** How `applyConstraints` and the
|
||||
empty-playable-set terminal state are modeled — owned by the
|
||||
track-switching constraints work, consumed here. (See
|
||||
[track-switching-model.md](../track-switching-model.md) → *Fitting the
|
||||
model to the track-switching behavior*.)
|
||||
- **Failure-window policy.** Threshold count / window length / cooldown
|
||||
duration for marking a CDN unhealthy. Empirical; lives with
|
||||
`network-resilience`'s circuit-breaker.
|
||||
- **CDN-id derivation configurability.** Origin-based `getCdnId` is the
|
||||
default; whether/where to expose a consumer override (config field vs.
|
||||
rule config view) is deferred until a non-origin case appears.
|
||||
- **Composition with content-steering.** Static redundant CDNs +
|
||||
dynamic steered host pool: does steering replace, intersect, or
|
||||
reprioritize the candidate CDNs? Open until content-steering lands; the
|
||||
reorderable `cdnPriority` list keeps it tractable (steering reorders it).
|
||||
- **Composition with content-steering.** Static redundant CDNs + a dynamic
|
||||
steered host pool: does steering replace, intersect, or reprioritize the
|
||||
candidate CDNs? Open until content-steering lands; the reorderable
|
||||
`cdnPriority` list keeps it tractable (steering reorders it).
|
||||
|
||||
### Resolved during sub-feature 1 implementation
|
||||
## Follow-up candidates
|
||||
|
||||
- **Signal shape** → a per-presentation ordered `cdnPriority` list (active
|
||||
= first-with-survivors, derived), not a single stored `activeCdn` value
|
||||
and not per-rendition. The list makes failover a pure constraint and
|
||||
composes with content-steering as a reorder. Matches the cross-type
|
||||
coherence requirement (one shared list).
|
||||
- **CDN identity** → URL origin (`getCdnId`); configurable derivation
|
||||
deferred.
|
||||
- **Manifest syntax / parser** → no change. Redundant variants already
|
||||
parse as separate per-CDN tracks; no `alternateUris` field needed.
|
||||
Known, intentionally-deferred refinements — none block the feature; worth
|
||||
tracking as a future effort:
|
||||
|
||||
- **No cooldown extension on re-failure.** A CDN's removal is scheduled when it
|
||||
first enters `failedCdns`; re-failing it mid-cooldown doesn't push the deadline
|
||||
out (set-membership watch). Fine for trip-on-first; revisit with a
|
||||
windowed/decaying health metric.
|
||||
- **Flapping.** A flaky-but-not-dead CDN can oscillate — trip → cooldown lapses →
|
||||
re-preferred (it's `cdnPriority[0]`) → fails again. No hysteresis or
|
||||
growing back-off on repeated trips.
|
||||
- **All-CDNs-down has no terminal state.** When every CDN is pruned the candidate
|
||||
set is empty and the prior pick is silently left in place; a distinct "nothing
|
||||
playable" state is unmodeled (shared with the constraints-pre-pass work).
|
||||
- **HTTP-status classification is coarse.** The trip fires on a thrown fetch or a
|
||||
non-OK media-playlist status; finer classification (5xx-with-body vs 4xx,
|
||||
segment-side status codes) is deferred to `network-resilience`.
|
||||
- **Retries below the trip.** `network-resilience` retry/backoff would sit under
|
||||
the fetch sites so the trip sees only post-retry terminal failures (fewer false
|
||||
trips). A refinement, not a dependency.
|
||||
- **`switchAudioTrack` config tidiness (cosmetic).** Audio spreads `...config`, so
|
||||
video-only ABR fields ride into the shared ranker harmlessly (audio has no
|
||||
`bandwidthState`). A cross-cutting-only shared config type would keep them out.
|
||||
|
||||
### Resolved during implementation
|
||||
|
||||
- **Signal shape** → a per-presentation ordered `cdnPriority` list (active =
|
||||
first-with-survivors, derived), not a stored `activeCdn` and not per-rendition.
|
||||
The list makes failover a pure constraint and composes with content-steering as
|
||||
a reorder; matches the one-shared-list cross-type coherence requirement.
|
||||
- **Failover detection** → **site-adds, behavior-expires**: fetch sites trip on
|
||||
the first terminal failure; `setupFailoverMonitor` expires after a cooldown.
|
||||
Self-contained — no `network-resilience` circuit-breaker dependency.
|
||||
- **CDN-id derivation** → configurable via the `getCdnId` engine config (origin
|
||||
default), threaded to all four CDN-id sites so the keys stay comparable.
|
||||
- **Constraints pre-pass** → built (`applyConstraints` + `constraints` config
|
||||
slot in `setupTrackSwitching`).
|
||||
- **Parse failures don't trip** — a 200 with an unparseable body is a content
|
||||
issue, not CDN unavailability; only fetch/non-OK failures trip.
|
||||
- **Manifest syntax / parser** → no change; redundant variants already parse as
|
||||
separate per-CDN tracks (no `alternateUris` field).
|
||||
- **Architecture** → constraint + scope in the track-switching model, not
|
||||
active-URI rotation in `resolveTrack`.
|
||||
|
||||
## Implementation surface
|
||||
|
||||
- **`packages/spf/src/media/utils/cdn.ts`** — `getCdnId(url)` (origin-based
|
||||
CDN identity) and `getOrderedCdnIds(presentation)` (distinct CDNs in
|
||||
manifest order; head = primary).
|
||||
- **`packages/spf/src/playback/behaviors/resolve-cdn-priority.ts`** —
|
||||
`resolveCdnPriority` behavior + `ResolveCdnPriorityState`. Machine reactor
|
||||
on `presentation-unresolved` ↔ `presentation-resolved`; owns the
|
||||
`cdnPriority` signal; publishes `getOrderedCdnIds(presentation)` (skipping
|
||||
the write when the CDN set is unchanged), clears on exit.
|
||||
- **`packages/spf/src/playback/behaviors/track-switching.ts`** —
|
||||
`preferActiveCdn` scope rule (soft filter on `cdnPriority`: narrow to the
|
||||
highest-priority CDN with surviving tracks), added to both variants'
|
||||
chains: `[filterByUserSelection, preferActiveCdn, rankByBandwidth]`.
|
||||
`SwitchableTrack` gains `url` (the rule's input).
|
||||
- **`packages/spf/src/playback/engines/hls/engine.ts` +
|
||||
`engine-audio-only.ts`** — `resolveCdnPriority` composed after
|
||||
`resolvePresentation`; `cdnPriority?: string[]` added to both engine state
|
||||
interfaces.
|
||||
- **`media/utils/cdn.ts`** — `getCdnId(url)` (origin-based default) + the
|
||||
`GetCdnId` type; `getOrderedCdnIds(presentation, getCdnId?)`;
|
||||
`addFailedCdn(failed, cdn)` (pure, idempotent dedup-append).
|
||||
- **`playback/behaviors/derive-cdn-priority.ts`** — `deriveCdnPriority` owns
|
||||
`cdnPriority` (publishes `getOrderedCdnIds` on resolve, skips unchanged
|
||||
writes, clears on exit).
|
||||
- **`playback/behaviors/setup-failover-monitor.ts`** — `setupFailoverMonitor`
|
||||
owns `failedCdns`; per-source, watches the set and schedules a cooldown
|
||||
removal per CDN, clears on src unload. Config `failover?: { cooldownMs }`.
|
||||
- **`playback/behaviors/resolve-track.ts`** — `failoverFetch(state, config)`
|
||||
decorates the media-playlist fetch (`fetchResolvableText`); a failed/non-OK
|
||||
fetch adds the CDN to `failedCdns` via `addFailedCdn`.
|
||||
- **`playback/behaviors/dom/setup-buffer-actors.ts`** — `failoverFetchBytes`
|
||||
decorates the per-type segment fetch (`trackedFetch` / `fetchStream`) the same way.
|
||||
- **`playback/behaviors/track-switching.ts`** — `preferActiveCdn` scope +
|
||||
`excludeFailedCdns` constraint (shared by the video + audio chains via the
|
||||
`CdnRuleConfig` view that carries `getCdnId`); `applyConstraints` pre-pass +
|
||||
`constraints` config slot; `SwitchableTrack` gains `url`.
|
||||
- **`playback/engines/hls/engine.ts` + `engine-audio-only.ts`** —
|
||||
`deriveCdnPriority` + `setupFailoverMonitor` composed after
|
||||
`resolvePresentation`; `failover?` + `getCdnId?` engine config; `cdnPriority?`
|
||||
+ `failedCdns?` engine state.
|
||||
- **`network/fetch.ts`** — `FetchText` type + `fetchResolvableText` default
|
||||
(fetch → reject on non-OK → text), the text analog of `FetchBytes`.
|
||||
|
||||
State signal: `cdnPriority?: string[]` (CDN origins, most-preferred first),
|
||||
owned by `resolveCdnPriority`, read optionally by `preferActiveCdn`.
|
||||
State signals: `cdnPriority?: string[]` (owned by `deriveCdnPriority`) and
|
||||
`failedCdns?: string[]` (owned by `setupFailoverMonitor`; tripped by the fetch
|
||||
sites, read by the `excludeFailedCdns` constraint).
|
||||
|
||||
## Verification
|
||||
|
||||
Sub-feature 1:
|
||||
- `media/utils/tests/cdn.test.ts` — `getCdnId` (origin; same/different host;
|
||||
scheme+port; unparseable fallback); `getOrderedCdnIds` (order; dedupe; single;
|
||||
unresolved → `[]`); `addFailedCdn` (append; order; idempotent same-reference).
|
||||
- `playback/behaviors/tests/derive-cdn-priority.test.ts` — publishes the
|
||||
manifest-ordered list; single-CDN; skips the write on a same-CDN swap; updates
|
||||
on reorder; clears on unload/destroy; re-publishes after reset.
|
||||
- `playback/behaviors/tests/setup-failover-monitor.test.ts` — a tripped CDN is
|
||||
removed once its cooldown lapses; independent per-CDN cooldowns; clears
|
||||
`failedCdns` on src unload; sensible cooldown default.
|
||||
- `playback/behaviors/tests/track-switching.test.ts` — `preferActiveCdn` (narrow
|
||||
to highest-priority surviving CDN; fall-through; cross-type coherence);
|
||||
`excludeFailedCdns` + `applyConstraints` (prune failed CDNs; order-independence;
|
||||
failover via the constraint).
|
||||
- `playback/engines/hls/tests/engine.test.ts` — integration: redundant-stream →
|
||||
`cdnPriority` = manifest order + pick on primary; reordering re-narrows;
|
||||
**auto-failover** (a failing media-playlist fetch trips the CDN and selection
|
||||
falls over to the backup, no external write); a **custom `getCdnId`** (keyed on
|
||||
a query param) honored across `cdnPriority`, the trip, and the constraint/scope
|
||||
end-to-end.
|
||||
- `playback/engines/hls/tests/failover-smoke.test.ts` — **gated live smoke test**
|
||||
(behind `VITE_FAILOVER_SMOKE`) against a real Mux `?redundant_streams=true`
|
||||
source: block one origin → trip → failover → recovery. Skipped by default.
|
||||
|
||||
- `media/utils/tests/cdn.test.ts` — `getCdnId` (origin extraction;
|
||||
same/different host; scheme+port; unparseable fallback);
|
||||
`getOrderedCdnIds` (distinct CDNs in order; dedupe; single-CDN;
|
||||
unresolved → `[]`).
|
||||
- `playback/behaviors/tests/resolve-cdn-priority.test.ts` — publishes the
|
||||
manifest-ordered CDN list; single-CDN source; skips the write when a
|
||||
resolved swap keeps the same CDNs; updates when the order changes; clears
|
||||
on src unload + on destroy; re-publishes after a src reset.
|
||||
- `playback/behaviors/tests/track-switching.test.ts` (`preferActiveCdn`
|
||||
block) — narrows to the highest-priority CDN overriding manifest track
|
||||
order; keeps the pick on the primary; falls through to the next CDN when
|
||||
the first has no survivors; falls through to all when none match; no-op
|
||||
when `cdnPriority` absent; re-picks reactively when the order changes
|
||||
(steering/failover seam); same scope applied to the audio chain
|
||||
(cross-type coherence).
|
||||
- `playback/engines/hls/tests/engine.test.ts` — integration: a
|
||||
redundant-stream presentation yields `cdnPriority` = manifest order and a
|
||||
video selection on the primary; reordering `cdnPriority` re-narrows and
|
||||
the selection follows.
|
||||
Deferred:
|
||||
|
||||
Out of scope / deferred:
|
||||
|
||||
- End-to-end + sandbox verification against a real Mux
|
||||
`?redundant_streams=true` source (needs a fixture).
|
||||
- All of sub-feature 2 (failover): blocked on the constraints phase +
|
||||
`network-resilience`.
|
||||
- E2e through the html player + real MSE — deferred; apps/e2e pages are generated
|
||||
from `media.ts`, so a redundant source would sweep into every generic + visual
|
||||
spec. The engine smoke test covers the round-trip (including recovery, which
|
||||
isn't observable at the player DOM).
|
||||
- A dedicated segment-failover integration test — the trip logic is unit-covered
|
||||
(`addFailedCdn`) and mirrors the tested media-playlist path.
|
||||
|
||||
## Related features
|
||||
|
||||
- **[track-switching-model.md](../track-switching-model.md)** *(governing
|
||||
model)* — multi-CDN is its canonical constraint + scope feature; the
|
||||
active-CDN scope is implemented against the rule chain it specifies.
|
||||
- **[network-resilience](./network-resilience.md)** *(hard prerequisite
|
||||
for failover)* — per-host circuit-breaker the failed-CDN constraint
|
||||
consumes.
|
||||
- **[network-resilience](./network-resilience.md)** *(optional refinement,
|
||||
not a prerequisite)* — retry/backoff below the trip would reduce false
|
||||
trips; failover ships self-contained.
|
||||
- **[content-steering](./content-steering.md)** — dynamic host-pool
|
||||
sibling; shares the active-pathway scope shape (`cdnPriority` as a
|
||||
reorderable reflected list — pathway priority as a sort key).
|
||||
- **[capability-probing](./capability-probing.md)** — shares the
|
||||
not-yet-built constraints pre-pass with the failed-CDN constraint.
|
||||
- **[capability-probing](./capability-probing.md)** — can reuse the
|
||||
constraints pre-pass (now built for the failed-CDN constraint).
|
||||
- **[live-stream-support](./live-stream-support.md)** *(not implemented)*
|
||||
— reload-loop re-resolution; `cdnPriority` is republished only when the
|
||||
CDN set changes.
|
||||
@@ -264,6 +307,10 @@ Out of scope / deferred:
|
||||
|
||||
## See also
|
||||
|
||||
- [multi-cdn-failover-prior-art.md](../multi-cdn-failover-prior-art.md) —
|
||||
how eight OSS players model CDN redundancy/failover, read against this
|
||||
design (two architectural families, content-steering convergence, and the
|
||||
prior art behind the 300s cooldown default + the open follow-ups)
|
||||
- [track-switching-model.md](../track-switching-model.md) — the rule
|
||||
model (constraints → soft filters → ranker) this feature composes into
|
||||
- [clusters.md § Selection resilience](./clusters.md#selection-resilience)
|
||||
@@ -271,8 +318,8 @@ Out of scope / deferred:
|
||||
`network-resilience`'s response-error handling
|
||||
- [clusters.md § Selection / filtering across clusters](./clusters.md#selection--filtering-across-clusters)
|
||||
— cluster G's role: alternate-CDN selection within the chosen track
|
||||
- [network-resilience.md](./network-resilience.md) — circuit-breaker
|
||||
foundation the failover tier consumes
|
||||
- [network-resilience.md](./network-resilience.md) — retry/backoff that
|
||||
would sit below the failover trip (optional refinement, not consumed today)
|
||||
- [SPF Epics Working Doc](https://www.notion.so/35f97a7f89d08123a13fecab1ca1cac4)
|
||||
— source material; epic #9 (Multi-CDN Failover)
|
||||
- [Mux Video — `?redundant_streams=true`](https://www.mux.com/docs/guides/play-back-on-multiple-cdns)
|
||||
|
||||
Reference in New Issue
Block a user