feat(spf): multi cdn failover (#1671)

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

internal/design/spf/features/multi-cdn-failover.md

@@ -1,7 +1,7 @@
 ---
 status: draft
-date: 2026-06-05
-definition: sketched
+date: 2026-06-08
+definition: implemented
 ---
 
 # Multi-CDN failover
@@ -22,12 +22,16 @@ Two sub-features, mapping onto the two rule kinds in that model:
    so video / audio / text resolve from one host. This is the
    *active-pathway scope* the track-switching model lists for
    multi-cdn-failover.
-2. **Failover** *(deferred)* — when requests to the active CDN fail too
-   often within a window, a **constraint** removes that CDN's tracks from
-   the candidate set during cooldown and the session behavior rotates the
-   active CDN. This is the *failed-CDN constraint* the model lists, and it
-   consumes [network-resilience](./network-resilience.md)'s per-host
-   circuit-breaker — its hard prerequisite.
+2. **Failover** *(implemented)* — **site-adds, behavior-expires.** Fetch
+   sites add a CDN to a `failedCdns` set on a failed fetch (the *trip*); a
+   shared **constraint** (`excludeFailedCdns`) prunes that CDN's tracks from
+   the candidate set so the scope falls to the next CDN; `setupFailoverMonitor`
+   removes the CDN once a cooldown lapses (the *expiry*) and the scope returns
+   to it. This is the *failed-CDN constraint* the model lists. It shipped
+   **self-contained** — a cooldown timer, *not*
+   [network-resilience](./network-resilience.md)'s circuit-breaker. Retries are
+   a future refinement that would sit *below* the trip (so it sees post-retry
+   terminal failures), not a prerequisite.
 
 A **Media-src feature** for sources that genuinely require failover, with
 a **Player feature** surface at the failover tier (customer-customizable
@@ -36,26 +40,30 @@ ambiguity reflects that split.
 
 ## Status
 
-- **Composition:** sub-feature 1 (sticky CDN pick) is implemented in
-  `createSimpleHlsEngine` and `createHlsAudioOnlyEngine`. The
-  `resolveCdnPriority` behavior owns the `cdnPriority` signal (the
-  manifest-ordered CDN list, most-preferred first); the `preferActiveCdn`
-  scope rule (shared by the video + audio chains in `track-switching`)
-  narrows candidates to the highest-priority CDN with surviving tracks.
-  Failover (sub-feature 2) is not implemented — no constraints pass, no
-  per-CDN failure tracking, no rotation.
-- **Definition depth:** sketched — sub-feature 1 has a populated
-  implementation surface + verification; sub-feature 2 stays at the
-  scope-and-constraints level pending its prerequisite.
-- **Hard prerequisite (failover only):**
-  [network-resilience](./network-resilience.md). The failed-CDN
-  constraint consumes the foundation's per-host circuit-breaker /
-  retry-exhaustion state. Sub-feature 1 has no such dependency — it's
-  pure selection over the already-parsed candidate set.
+- **Composition:** both sub-features are implemented in
+  `createSimpleHlsEngine` and the audio-only engine. `deriveCdnPriority`
+  owns the `cdnPriority` signal (manifest-ordered CDN list); the
+  `preferActiveCdn` scope rule narrows candidates to the highest-priority CDN
+  with surviving tracks (shared by the video + audio chains). For failover:
+  the `excludeFailedCdns` constraint prunes tracks whose CDN is in the
+  `failedCdns` set; fetch sites trip a CDN into `failedCdns` on a failed fetch
+  (`failoverFetch` for media playlists in `resolve-track`, `failoverFetchBytes`
+  for segments in `setup-buffer-actors`); `setupFailoverMonitor` owns `failedCdns`
+  and removes each CDN once its cooldown lapses.
+- **Definition depth:** implemented.
+- **Detection (self-contained, no hard prerequisite):** a CDN trips on the
+  **first terminal fetch failure** (network error or non-OK status); cooldown
+  is the only back-off. `setupFailoverMonitor` is a per-source cooldown timer,
+  *not* a circuit-breaker imported from
+  [network-resilience](./network-resilience.md). Retries are a future
+  refinement *below* the trip (so it would observe only post-retry terminal
+  failures), not a dependency — this feature ships without
+  `network-resilience` existing.
 - **Governing model:** [track-switching-model.md](../track-switching-model.md)
-  — multi-CDN is the canonical *constraint + scope* feature there. The
-  active-CDN scope is a soft filter in the rule chain; the failed-CDN
-  constraint is a hard filter in the (not-yet-built) constraints pre-pass.
+  — multi-CDN is the canonical *constraint + scope* feature there. Both halves
+  are now built: the active-CDN scope (soft filter in the rule chain) and the
+  failed-CDN constraint (hard filter in the constraints pre-pass —
+  `applyConstraints` + the `constraints` config slot now exist).
 
 ## How redundant streams are modeled
 
@@ -65,7 +73,8 @@ redundant-streams manifest lists each rendition once per CDN (duplicate
 existing parser already emits one `Track` per entry with a unique id and
 its own absolute `url`. So the candidate set for each type *already*
 contains one variant per CDN. CDN identity is derived from each track
-URL's origin (`getCdnId`); the set of CDNs is published as a single
+URL's origin (`getCdnId`, overridable via engine config — e.g. to key on
+Mux's `cdn=` query param); the set of CDNs is published as a single
 per-presentation ordered signal (`cdnPriority`, most-preferred first —
 mirroring HLS content steering's `PATHWAY-PRIORITY`). The *active* CDN is
 not stored: the scope derives it as the highest-priority `cdnPriority`
@@ -74,38 +83,39 @@ CDN-tagged track id means `resolveTrack` / segment loading fetch from that
 CDN with no further plumbing.
 
 This list shape is what makes failover fall out cleanly: the failed-CDN
-constraint (sub-feature 2) prunes a cooled-down CDN's tracks, so
-"first-with-survivors" moves to the next CDN automatically and returns to
-the primary when it recovers — no reactive rewrite of an "active" value.
-Content steering, likewise, just reorders `cdnPriority` (pathway priority
-as a sort key).
+constraint prunes a cooled-down CDN's tracks, so "first-with-survivors"
+moves to the next CDN automatically and returns to the primary when it
+recovers — no reactive rewrite of an "active" value. Content steering,
+likewise, just reorders `cdnPriority` (pathway priority as a sort key).
 
-This is why sub-feature 1 needed **no parser change** and no new data
-shape — only a selection behavior and a scope rule over existing tracks.
+This is why the feature needed **no parser change** and no new data shape
+— only selection behaviors, a scope rule, and a constraint over existing
+tracks, plus a fetch decorator that records failures into `failedCdns`.
 
 ## Phases of complexity
 
 | Phase | Sub-feature | Kind | What | State |
 |---|---|---|---|---|
-| Sticky CDN pick | 1 | scope | `resolveCdnPriority` publishes the manifest-ordered CDN list (`cdnPriority`); `preferActiveCdn` narrows every type's candidates to the highest-priority CDN with surviving tracks, falling through when nothing matches. Shared list → all types on one CDN | **Implemented** |
-| Failed-CDN constraint | 2 | constraint | A constraint in the track-switching constraints pre-pass removes a cooled-down CDN's tracks from the candidate set. The scope then picks the next `cdnPriority` entry automatically. Requires the generic constraints phase (track-switching-model "Phase 2") to be built first | Deferred |
-| Per-CDN failure tracking | 2 | — | Count per-CDN fetch failures within a window; mark a CDN unhealthy / in cooldown. Consumes `network-resilience`'s circuit-breaker | Deferred (prereq) |
-| CDN priority override / steering | 2 | scope | Reorder `cdnPriority` to bias the pick (region-preferred, weighted, or content-steering's pathway priority). No reactive "active" rewrite needed — the order *is* the policy | Deferred |
-| Customer CDN-id derivation | 2 | config | Pluggable `getCdnId` for non-origin identity. The origin-based default is built in; a config seam is anticipated | Deferred |
+| Sticky CDN pick | 1 | scope | `deriveCdnPriority` publishes the manifest-ordered CDN list (`cdnPriority`); `preferActiveCdn` narrows every type's candidates to the highest-priority CDN with surviving tracks, falling through when nothing matches. Shared list → all types on one CDN | **Implemented** |
+| Constraints pre-pass | 2 | — | `applyConstraints` + the `constraints` config slot in `setupTrackSwitching` (the hard-filter pre-pass that runs before the rule chain). Reusable by capability-probing | **Implemented** |
+| Failed-CDN constraint | 2 | constraint | `excludeFailedCdns` prunes tracks whose CDN ∈ `failedCdns`; the scope falls to the next `cdnPriority` entry and snaps back on recovery | **Implemented** |
+| Per-CDN failure tracking | 2 | — | **Site-adds, behavior-expires**: fetch sites trip a CDN into `failedCdns` on a failed fetch (`failoverFetch` / `failoverFetchBytes`); `setupFailoverMonitor` expires it after a cooldown. Self-contained (trip-on-first-failure + cooldown), not a `network-resilience` circuit-breaker | **Implemented** |
+| Customer CDN-id derivation | 2 | config | Pluggable `getCdnId` (engine config) for non-origin identity, threaded to all four CDN-id sites; origin-based default | **Implemented** |
+| CDN priority override / steering | 2 | scope | Reorder `cdnPriority` to bias the pick (region-preferred, weighted, or content-steering's pathway priority). No reactive "active" rewrite needed — the order *is* the policy | Deferred (content-steering) |
 
 ## What's in scope vs out of scope
 
-**In scope:**
-- Sticky per-presentation CDN selection (sub-feature 1, done)
-- Failover via a track-switching constraint + active-CDN rotation
-  (sub-feature 2)
-- Per-CDN health derived from `network-resilience`'s circuit-breaker
-- Customer-configurable CDN-id derivation / rotation policy
+**In scope (all implemented):**
+- Sticky per-presentation CDN selection
+- Failover via the failed-CDN constraint + derived active-CDN rotation
+- Self-contained per-CDN failure tracking (trip-on-first-failure + cooldown)
+- Customer-configurable CDN-id derivation (`getCdnId`)
 
 **Out of scope (separate cluster G sister features):**
-- **[network-resilience](./network-resilience.md)** *(foundation,
-  prerequisite for failover)* — retry + backoff + circuit-breaker.
-  Multi-CDN consumes; doesn't reimplement.
+- **[network-resilience](./network-resilience.md)** *(optional future
+  refinement, not a prerequisite)* — retry + backoff + error-classification.
+  Would sit *below* the failover trip to reduce false trips; failover ships
+  without it.
 - **[content-steering](./content-steering.md)** — HLS content-steering
   protocol (server-advertised, dynamically-updated host pool). Different
   mechanism than static redundant streams, but the *same* active-pathway
@@ -129,7 +139,7 @@ shape — only a selection behavior and a scope rule over existing tracks.
   video and audio chains reference the *same* `preferActiveCdn` definition
   reading the *same* list, so they agree on the CDN even if their per-type
   track arrays differ. (The doc's earlier per-rendition lean is superseded.)
-- **`cdnPriority` writer composition.** `resolveCdnPriority` is the sole
+- **`cdnPriority` writer composition.** `deriveCdnPriority` is the sole
   writer today (publishes the manifest order). Failover needs no second
   writer — the failed-CDN constraint prunes tracks and the scope re-derives
   the active CDN. Content-steering would *reorder* `cdnPriority` (still a
@@ -140,120 +150,153 @@ shape — only a selection behavior and a scope rule over existing tracks.
   the primary. This is why the array beats a single reactive `activeCdn`
   value — the failed-set information is applied once (in the constraint),
   not duplicated into an active-value rewrite.
-- **Constraints phase is a shared prerequisite.** The failed-CDN
-  constraint can't land until the generic constraints pre-pass
-  (`applyConstraints` + the `constraints` config field + empty-playable-set
-  terminal state) is built into `setupTrackSwitching`. The seam exists
-  (`candidateSet` computed); the machinery does not. capability-probing
-  shares this prerequisite.
+- **Constraints pre-pass (now built).** The failed-CDN constraint runs in
+  `setupTrackSwitching`'s `applyConstraints` pre-pass — the `constraints`
+  config slot, applied to `candidateSet` before the rule chain.
+  capability-probing can reuse it. One piece is deliberately *not* built: a
+  terminal "everything pruned" state — today an all-CDNs-failed candidate set
+  is empty and the prior pick is left in place (see *Follow-up candidates*).
 - **Live + multi-CDN.** During live playback the reload loop re-resolves
-  the presentation; `resolveCdnPriority` re-publishes only when the CDN set
+  the presentation; `deriveCdnPriority` re-publishes only when the CDN set
   changes (idempotent for a stable manifest). Cross-feature with
   [live-stream-support](./live-stream-support.md) (not yet implemented).
-- **Priority state across source changes.** `cdnPriority` tears down with
-  the source via the resolved/unresolved cascade (per
-  [source-replacement](./source-replacement.md)); per-host circuit-breaker
-  state in `network-resilience` may outlive a source.
+- **Failover state is per-source.** Both `cdnPriority` and `failedCdns` tear
+  down with the source via the resolved/unresolved cascade (per
+  [source-replacement](./source-replacement.md)) — `setupFailoverMonitor` clears
+  `failedCdns` and its cooldown timers on unload, so no failover state leaks
+  across sources. (If `network-resilience` lands later, its per-host state may
+  choose to outlive a source — that's its call, not failover's.)
 
 ## Open questions
 
-- **Constraints-phase shape.** How `applyConstraints` and the
-  empty-playable-set terminal state are modeled — owned by the
-  track-switching constraints work, consumed here. (See
-  [track-switching-model.md](../track-switching-model.md) → *Fitting the
-  model to the track-switching behavior*.)
-- **Failure-window policy.** Threshold count / window length / cooldown
-  duration for marking a CDN unhealthy. Empirical; lives with
-  `network-resilience`'s circuit-breaker.
-- **CDN-id derivation configurability.** Origin-based `getCdnId` is the
-  default; whether/where to expose a consumer override (config field vs.
-  rule config view) is deferred until a non-origin case appears.
-- **Composition with content-steering.** Static redundant CDNs +
-  dynamic steered host pool: does steering replace, intersect, or
-  reprioritize the candidate CDNs? Open until content-steering lands; the
-  reorderable `cdnPriority` list keeps it tractable (steering reorders it).
+- **Composition with content-steering.** Static redundant CDNs + a dynamic
+  steered host pool: does steering replace, intersect, or reprioritize the
+  candidate CDNs? Open until content-steering lands; the reorderable
+  `cdnPriority` list keeps it tractable (steering reorders it).
 
-### Resolved during sub-feature 1 implementation
+## Follow-up candidates
 
-- **Signal shape** → a per-presentation ordered `cdnPriority` list (active
-  = first-with-survivors, derived), not a single stored `activeCdn` value
-  and not per-rendition. The list makes failover a pure constraint and
-  composes with content-steering as a reorder. Matches the cross-type
-  coherence requirement (one shared list).
-- **CDN identity** → URL origin (`getCdnId`); configurable derivation
-  deferred.
-- **Manifest syntax / parser** → no change. Redundant variants already
-  parse as separate per-CDN tracks; no `alternateUris` field needed.
+Known, intentionally-deferred refinements — none block the feature; worth
+tracking as a future effort:
+
+- **No cooldown extension on re-failure.** A CDN's removal is scheduled when it
+  first enters `failedCdns`; re-failing it mid-cooldown doesn't push the deadline
+  out (set-membership watch). Fine for trip-on-first; revisit with a
+  windowed/decaying health metric.
+- **Flapping.** A flaky-but-not-dead CDN can oscillate — trip → cooldown lapses →
+  re-preferred (it's `cdnPriority[0]`) → fails again. No hysteresis or
+  growing back-off on repeated trips.
+- **All-CDNs-down has no terminal state.** When every CDN is pruned the candidate
+  set is empty and the prior pick is silently left in place; a distinct "nothing
+  playable" state is unmodeled (shared with the constraints-pre-pass work).
+- **HTTP-status classification is coarse.** The trip fires on a thrown fetch or a
+  non-OK media-playlist status; finer classification (5xx-with-body vs 4xx,
+  segment-side status codes) is deferred to `network-resilience`.
+- **Retries below the trip.** `network-resilience` retry/backoff would sit under
+  the fetch sites so the trip sees only post-retry terminal failures (fewer false
+  trips). A refinement, not a dependency.
+- **`switchAudioTrack` config tidiness (cosmetic).** Audio spreads `...config`, so
+  video-only ABR fields ride into the shared ranker harmlessly (audio has no
+  `bandwidthState`). A cross-cutting-only shared config type would keep them out.
+
+### Resolved during implementation
+
+- **Signal shape** → a per-presentation ordered `cdnPriority` list (active =
+  first-with-survivors, derived), not a stored `activeCdn` and not per-rendition.
+  The list makes failover a pure constraint and composes with content-steering as
+  a reorder; matches the one-shared-list cross-type coherence requirement.
+- **Failover detection** → **site-adds, behavior-expires**: fetch sites trip on
+  the first terminal failure; `setupFailoverMonitor` expires after a cooldown.
+  Self-contained — no `network-resilience` circuit-breaker dependency.
+- **CDN-id derivation** → configurable via the `getCdnId` engine config (origin
+  default), threaded to all four CDN-id sites so the keys stay comparable.
+- **Constraints pre-pass** → built (`applyConstraints` + `constraints` config
+  slot in `setupTrackSwitching`).
+- **Parse failures don't trip** — a 200 with an unparseable body is a content
+  issue, not CDN unavailability; only fetch/non-OK failures trip.
+- **Manifest syntax / parser** → no change; redundant variants already parse as
+  separate per-CDN tracks (no `alternateUris` field).
 - **Architecture** → constraint + scope in the track-switching model, not
   active-URI rotation in `resolveTrack`.
 
 ## Implementation surface
 
-- **`packages/spf/src/media/utils/cdn.ts`** — `getCdnId(url)` (origin-based
-  CDN identity) and `getOrderedCdnIds(presentation)` (distinct CDNs in
-  manifest order; head = primary).
-- **`packages/spf/src/playback/behaviors/resolve-cdn-priority.ts`** —
-  `resolveCdnPriority` behavior + `ResolveCdnPriorityState`. Machine reactor
-  on `presentation-unresolved` ↔ `presentation-resolved`; owns the
-  `cdnPriority` signal; publishes `getOrderedCdnIds(presentation)` (skipping
-  the write when the CDN set is unchanged), clears on exit.
-- **`packages/spf/src/playback/behaviors/track-switching.ts`** —
-  `preferActiveCdn` scope rule (soft filter on `cdnPriority`: narrow to the
-  highest-priority CDN with surviving tracks), added to both variants'
-  chains: `[filterByUserSelection, preferActiveCdn, rankByBandwidth]`.
-  `SwitchableTrack` gains `url` (the rule's input).
-- **`packages/spf/src/playback/engines/hls/engine.ts` +
-  `engine-audio-only.ts`** — `resolveCdnPriority` composed after
-  `resolvePresentation`; `cdnPriority?: string[]` added to both engine state
-  interfaces.
+- **`media/utils/cdn.ts`** — `getCdnId(url)` (origin-based default) + the
+  `GetCdnId` type; `getOrderedCdnIds(presentation, getCdnId?)`;
+  `addFailedCdn(failed, cdn)` (pure, idempotent dedup-append).
+- **`playback/behaviors/derive-cdn-priority.ts`** — `deriveCdnPriority` owns
+  `cdnPriority` (publishes `getOrderedCdnIds` on resolve, skips unchanged
+  writes, clears on exit).
+- **`playback/behaviors/setup-failover-monitor.ts`** — `setupFailoverMonitor`
+  owns `failedCdns`; per-source, watches the set and schedules a cooldown
+  removal per CDN, clears on src unload. Config `failover?: { cooldownMs }`.
+- **`playback/behaviors/resolve-track.ts`** — `failoverFetch(state, config)`
+  decorates the media-playlist fetch (`fetchResolvableText`); a failed/non-OK
+  fetch adds the CDN to `failedCdns` via `addFailedCdn`.
+- **`playback/behaviors/dom/setup-buffer-actors.ts`** — `failoverFetchBytes`
+  decorates the per-type segment fetch (`trackedFetch` / `fetchStream`) the same way.
+- **`playback/behaviors/track-switching.ts`** — `preferActiveCdn` scope +
+  `excludeFailedCdns` constraint (shared by the video + audio chains via the
+  `CdnRuleConfig` view that carries `getCdnId`); `applyConstraints` pre-pass +
+  `constraints` config slot; `SwitchableTrack` gains `url`.
+- **`playback/engines/hls/engine.ts` + `engine-audio-only.ts`** —
+  `deriveCdnPriority` + `setupFailoverMonitor` composed after
+  `resolvePresentation`; `failover?` + `getCdnId?` engine config; `cdnPriority?`
+  + `failedCdns?` engine state.
+- **`network/fetch.ts`** — `FetchText` type + `fetchResolvableText` default
+  (fetch → reject on non-OK → text), the text analog of `FetchBytes`.
 
-State signal: `cdnPriority?: string[]` (CDN origins, most-preferred first),
-owned by `resolveCdnPriority`, read optionally by `preferActiveCdn`.
+State signals: `cdnPriority?: string[]` (owned by `deriveCdnPriority`) and
+`failedCdns?: string[]` (owned by `setupFailoverMonitor`; tripped by the fetch
+sites, read by the `excludeFailedCdns` constraint).
 
 ## Verification
 
-Sub-feature 1:
+- `media/utils/tests/cdn.test.ts` — `getCdnId` (origin; same/different host;
+  scheme+port; unparseable fallback); `getOrderedCdnIds` (order; dedupe; single;
+  unresolved → `[]`); `addFailedCdn` (append; order; idempotent same-reference).
+- `playback/behaviors/tests/derive-cdn-priority.test.ts` — publishes the
+  manifest-ordered list; single-CDN; skips the write on a same-CDN swap; updates
+  on reorder; clears on unload/destroy; re-publishes after reset.
+- `playback/behaviors/tests/setup-failover-monitor.test.ts` — a tripped CDN is
+  removed once its cooldown lapses; independent per-CDN cooldowns; clears
+  `failedCdns` on src unload; sensible cooldown default.
+- `playback/behaviors/tests/track-switching.test.ts` — `preferActiveCdn` (narrow
+  to highest-priority surviving CDN; fall-through; cross-type coherence);
+  `excludeFailedCdns` + `applyConstraints` (prune failed CDNs; order-independence;
+  failover via the constraint).
+- `playback/engines/hls/tests/engine.test.ts` — integration: redundant-stream →
+  `cdnPriority` = manifest order + pick on primary; reordering re-narrows;
+  **auto-failover** (a failing media-playlist fetch trips the CDN and selection
+  falls over to the backup, no external write); a **custom `getCdnId`** (keyed on
+  a query param) honored across `cdnPriority`, the trip, and the constraint/scope
+  end-to-end.
+- `playback/engines/hls/tests/failover-smoke.test.ts` — **gated live smoke test**
+  (behind `VITE_FAILOVER_SMOKE`) against a real Mux `?redundant_streams=true`
+  source: block one origin → trip → failover → recovery. Skipped by default.
 
-- `media/utils/tests/cdn.test.ts` — `getCdnId` (origin extraction;
-  same/different host; scheme+port; unparseable fallback);
-  `getOrderedCdnIds` (distinct CDNs in order; dedupe; single-CDN;
-  unresolved → `[]`).
-- `playback/behaviors/tests/resolve-cdn-priority.test.ts` — publishes the
-  manifest-ordered CDN list; single-CDN source; skips the write when a
-  resolved swap keeps the same CDNs; updates when the order changes; clears
-  on src unload + on destroy; re-publishes after a src reset.
-- `playback/behaviors/tests/track-switching.test.ts` (`preferActiveCdn`
-  block) — narrows to the highest-priority CDN overriding manifest track
-  order; keeps the pick on the primary; falls through to the next CDN when
-  the first has no survivors; falls through to all when none match; no-op
-  when `cdnPriority` absent; re-picks reactively when the order changes
-  (steering/failover seam); same scope applied to the audio chain
-  (cross-type coherence).
-- `playback/engines/hls/tests/engine.test.ts` — integration: a
-  redundant-stream presentation yields `cdnPriority` = manifest order and a
-  video selection on the primary; reordering `cdnPriority` re-narrows and
-  the selection follows.
+Deferred:
 
-Out of scope / deferred:
-
-- End-to-end + sandbox verification against a real Mux
-  `?redundant_streams=true` source (needs a fixture).
-- All of sub-feature 2 (failover): blocked on the constraints phase +
-  `network-resilience`.
+- E2e through the html player + real MSE — deferred; apps/e2e pages are generated
+  from `media.ts`, so a redundant source would sweep into every generic + visual
+  spec. The engine smoke test covers the round-trip (including recovery, which
+  isn't observable at the player DOM).
+- A dedicated segment-failover integration test — the trip logic is unit-covered
+  (`addFailedCdn`) and mirrors the tested media-playlist path.
 
 ## Related features
 
 - **[track-switching-model.md](../track-switching-model.md)** *(governing
   model)* — multi-CDN is its canonical constraint + scope feature; the
   active-CDN scope is implemented against the rule chain it specifies.
-- **[network-resilience](./network-resilience.md)** *(hard prerequisite
-  for failover)* — per-host circuit-breaker the failed-CDN constraint
-  consumes.
+- **[network-resilience](./network-resilience.md)** *(optional refinement,
+  not a prerequisite)* — retry/backoff below the trip would reduce false
+  trips; failover ships self-contained.
 - **[content-steering](./content-steering.md)** — dynamic host-pool
   sibling; shares the active-pathway scope shape (`cdnPriority` as a
   reorderable reflected list — pathway priority as a sort key).
-- **[capability-probing](./capability-probing.md)** — shares the
-  not-yet-built constraints pre-pass with the failed-CDN constraint.
+- **[capability-probing](./capability-probing.md)** — can reuse the
+  constraints pre-pass (now built for the failed-CDN constraint).
 - **[live-stream-support](./live-stream-support.md)** *(not implemented)*
   — reload-loop re-resolution; `cdnPriority` is republished only when the
   CDN set changes.
@@ -264,6 +307,10 @@ Out of scope / deferred:
 
 ## See also
 
+- [multi-cdn-failover-prior-art.md](../multi-cdn-failover-prior-art.md) —
+  how eight OSS players model CDN redundancy/failover, read against this
+  design (two architectural families, content-steering convergence, and the
+  prior art behind the 300s cooldown default + the open follow-ups)
 - [track-switching-model.md](../track-switching-model.md) — the rule
   model (constraints → soft filters → ranker) this feature composes into
 - [clusters.md § Selection resilience](./clusters.md#selection-resilience)
@@ -271,8 +318,8 @@ Out of scope / deferred:
   `network-resilience`'s response-error handling
 - [clusters.md § Selection / filtering across clusters](./clusters.md#selection--filtering-across-clusters)
   — cluster G's role: alternate-CDN selection within the chosen track
-- [network-resilience.md](./network-resilience.md) — circuit-breaker
-  foundation the failover tier consumes
+- [network-resilience.md](./network-resilience.md) — retry/backoff that
+  would sit below the failover trip (optional refinement, not consumed today)
 - [SPF Epics Working Doc](https://www.notion.so/35f97a7f89d08123a13fecab1ca1cac4)
   — source material; epic #9 (Multi-CDN Failover)
 - [Mux Video — `?redundant_streams=true`](https://www.mux.com/docs/guides/play-back-on-multiple-cdns)

internal/design/spf/multi-cdn-failover-prior-art.md

@@ -0,0 +1,209 @@
+---
+status: draft
+date: 2026-06-12
+---
+
+# Multi-CDN failover: prior art
+
+> A survey of how eight open-source players and playback engines model CDN
+> redundancy and failover, read against SPF's design. It backs
+> [features/multi-cdn-failover.md](./features/multi-cdn-failover.md),
+> [features/content-steering.md](./features/content-steering.md), and
+> [features/network-resilience.md](./features/network-resilience.md), and
+> informs the constraint+scope framing in
+> [track-switching-model.md](./track-switching-model.md). Iterate freely —
+> this is a reference frame, not a status record.
+
+## Scope
+
+The research question: when a source publishes the same content on more than
+one CDN (e.g. Mux Video's `?redundant_streams=true`), how do existing players
+(a) represent that redundancy, (b) decide a CDN has failed, (c) recover, and
+(d) compose with HLS/DASH Content Steering?
+
+Surveyed: video.js v8 http-streaming (VHS), hls.js, dash.js, shaka-player,
+rx-player, Media3/ExoPlayer, VLC, OSMF. Citations are against each project's
+repository (see *Sources*) at survey time and may drift — verify before
+quoting as current.
+
+## The two architectural families
+
+Every player that does real failover falls into one of two camps for **how
+redundancy is modeled**. This is the axis SPF is most interesting against.
+
+**Family A — URL/URI rotation at the resource level.** The segment (or
+BaseURL) carries *multiple URIs*; failover picks a different URI for that
+resource. Redundancy lives *inside* the resource.
+
+- [**shaka**](https://github.com/shaka-project/shaka-player) — segments expose
+  `getUris()` → an array; `NetworkingEngine` rotates via
+  `index = request.attempt % request.uris.length`
+  (`lib/net/networking_engine.js`).
+- [**dash.js**](https://github.com/Dash-Industry-Forum/dash.js) — multiple
+  `<BaseURL>` per MPD node; `BaseURLController` selects per node
+  (`src/streaming/controllers/BaseURLController.js`).
+- [**rx-player**](https://github.com/canalplus/rx-player) — `ICdnMetadata[]`
+  per Representation; a `CdnPrioritizer` picks among them
+  (`src/core/fetchers/cdn_prioritizer.ts`).
+- [**media3 (DASH path)**](https://github.com/androidx/media) —
+  `BaseUrlExclusionList` filters `BaseUrl` objects
+  (`libraries/exoplayer_dash/.../BaseUrlExclusionList.java`).
+- [**VLC**](https://github.com/videolan/vlc) — parses multiple `<BaseURL>` but
+  *only ever uses `baseUrls.front()`*
+  (`modules/demux/adaptive/playlist/BasePlaylist.cpp`) — decorative.
+
+**Family B — pathway/ladder selection.** Redundant variants are *separate
+playlists/levels* grouped by a pathway id; failover switches which whole
+ladder is active. Redundancy lives *above* the resource.
+
+- [**hls.js**](https://github.com/video-dev/hls.js) — redundant streams become
+  separate `Level`s; pathway ids auto-assigned `"."`, `".."`, `"..."`; the
+  `ContentSteeringController` swaps ladders
+  (`src/controller/level-controller.ts`,
+  `src/controller/content-steering-controller.ts`).
+- [**VHS**](https://github.com/videojs/http-streaming) — separate playlists
+  keyed by `PATHWAY-ID || serviceLocation`; exclusion + steering pathway switch
+  (`src/playlist-controller.js`).
+- **media3 (HLS path)** — `HlsRedundantGroup` maps pathway ids → playlist URLs
+  (`libraries/exoplayer_hls/.../playlist/HlsRedundantGroup.java`).
+
+### Where SPF sits
+
+Family B *in spirit* — redundant variants parse as separate candidate tracks,
+failover is a ladder-level concept — but expressed **declaratively as a
+selection constraint in the track-switching rule chain** rather than
+imperative "switch pathway" controller logic. None of the eight do this. The
+closest mental model is hls.js / media3 "pathway = a complete ladder," but
+they all *imperatively* reassign level/playlist indices on switch
+(`reassignFragmentLevelIndexes()`, `switchPathway()`). SPF makes failover fall
+out of the same `candidateSet` computed that already does ABR/track selection:
+"prune the failed CDN's tracks, scope falls to the next entry" is structurally
+the same operation as any other filter rule. That is the genuinely novel
+framing, and it is why the feature needed no parser change and no dedicated
+failover state machine — see
+[features/multi-cdn-failover.md § How redundant streams are modeled](./features/multi-cdn-failover.md).
+
+## Comparison by axis
+
+| Player | Model | Trip trigger | Recovery | CDN identity | Scope |
+|---|---|---|---|---|---|
+| **SPF (v10)** | Separate candidate tracks; failover = selection constraint | **First terminal fetch failure** (playlist or segment), no threshold | 300s cooldown expiry (config) | **URL origin, configurable `getCdnId`** | Per-presentation shared list |
+| **VHS** | Separate playlists by pathway | First failure → temporal exclude; error *count* → permanent (`maxPlaylistRetries`) | Temporal expiry; permanent past threshold; last-rendition fallback clears others | `PATHWAY-ID \|\| serviceLocation` | Per-playlist + steering |
+| **hls.js** | Separate levels by pathway | Cumulative **error threshold/retries**, then penalty box | **300s** penalty cooldown | `PATHWAY-ID` (auto-dotted) | Per-level + per-pathway |
+| **dash.js** | Multiple BaseURLs per node | **First failure** → blacklist serviceLocation | Blacklist expiry; **indefinite by default** unless steering TTL | `serviceLocation` + DVB priority/weight | Per-node, sticky-cached |
+| **shaka** | Multiple URIs per segment | Per-request retry rotation (modulo) | **Stateless** — each segment restarts at `uris[0]`; steering ban = 60s | serviceLocation / pathway | Per-request |
+| **rx-player** | `ICdnMetadata[]` per Representation | First failure → downgrade + per-CDN retry counter → permanent at maxRetry | ~60s downgrade¹; `priorityChange` event pivots mid-backoff | `id` (≈ serviceLocation), baseUrl fallback | Per-segment, global prioritizer |
+| **media3** | DASH: BaseUrls; HLS: redundant groups | **First failure, selective HTTP codes** (403/404/410/416/500/503) | **Asymmetric: 300s location / 60s track** | serviceLocation / pathway | 3 layers: chunk / track / location |
+| **VLC** | Parses BaseURLs, uses only first | **No inter-CDN failover** — HTTP 3xx redirects only (max 3) | None | hostname/port (implicit) | Per-chunk redirect |
+| **OSMF** | `serverBaseURLs[]` parsed but unused | **None** — retries same URL on timeout | None | manifest baseURL | Per-fragment |
+
+¹ rx-player's `DEFAULT_CDN_DOWNGRADE_TIME` reads `60` ms in `default_config.ts`
+— most likely a typo for seconds; its tests use `5000`. Treat the intent as
+"seconds," the value as unverified.
+
+## Content Steering: near-universal convergence
+
+The modern players implement it almost identically, which validates SPF's
+choice to name its ordered list `cdnPriority` after HLS Content Steering's
+`PATHWAY-PRIORITY`:
+
+- **hls.js, VHS, dash.js, shaka, media3** all parse a steering manifest with
+  `PATHWAY-PRIORITY` / `SERVICE-LOCATION-PRIORITY`, `PATHWAY-CLONES` (HOST /
+  PARAMS URI rewriting), TTL-based reload, and `_HLS_pathway` / `_HLS_throughput`
+  (or `_DASH_*`) query hints. Steering **reorders** the priority list and can
+  **synthesize** new CDN pathways via clones.
+- **rx-player** built `CdnPrioritizer` *specifically* as the steering
+  substrate but hasn't wired steering in — "waiting for the spec to be
+  standardized and relied on in the wild" (`cdn_prioritizer.ts` class comment).
+- **VLC, OSMF** — absent.
+
+The industry consensus — "steering reorders the priority list and clone-rewrites
+URIs" — is exactly the design [features/multi-cdn-failover.md](./features/multi-cdn-failover.md)
+anticipates: `cdnPriority` is a reorderable list a steering behavior would
+write (pathway priority as a sort key). The one piece everyone else has that
+SPF would add later: **pathway clones** (HOST/PARAMS rewriting to synthesize
+CDNs absent from the manifest). SPF's configurable `getCdnId` is the seam where
+clone-rewritten URLs would need consistent identity. Tracked in
+[features/content-steering.md](./features/content-steering.md).
+
+## Ideas worth adopting
+
+Mapped to the *Follow-up candidates* in
+[features/multi-cdn-failover.md](./features/multi-cdn-failover.md):
+
+1. **Asymmetric cooldowns (media3, hls.js).** media3 uses **300s for
+   location/CDN exclusion vs 60s for track exclusion** — the principle being
+   "CDN failures are infrastructure problems (longer) than bitrate issues
+   (shorter)." This is the prior art behind bumping SPF's default cooldown
+   from 30s to **300s**. The next refinement — cooldown-extension on re-failure
+   — is what hls.js's penalty box and media3's exclusion already do.
+
+2. **rx-player's `priorityChange` event** is the most elegant recovery design:
+   when a downgraded CDN's cooldown expires it *interrupts an in-flight backoff
+   wait* and pivots immediately. Push-driven recovery, not "next request happens
+   to re-check." Relevant if SPF ever wants recovery faster than the next
+   natural fetch.
+
+3. **media3's `LoadErrorHandlingPolicy` / `FallbackSelection` /
+   `FallbackOptions`** is the cleanest *abstraction*: a pluggable policy that,
+   given `FallbackOptions(numberOfLocations, numberOfTracks)`, returns either a
+   location-fallback or a track-fallback with a duration — and **prefers
+   location fallback over track fallback** when both are available. It cleanly
+   separates "this is a CDN problem" from "this is a quality problem." SPF's
+   `applyConstraints` pre-pass is the analogous seam; a mature policy layer on
+   top would look like this.
+
+4. **Selective HTTP-status classification (media3; VHS's 410→permanent,
+   429→Retry-After).** Directly addresses the *HTTP-status classification is
+   coarse* follow-up. media3 is the reference for which codes should trip vs
+   retry.
+
+5. **VHS's last-rendition fallback** — when excluding the final playlist,
+   proactively clear other temporal exclusions instead of erroring — is exactly
+   the *all-CDNs-down has no terminal state* gap. Their answer: don't have a
+   terminal state; re-admit everyone and retry rather than hard-fail.
+
+6. **shaka's stateless per-request rotation** is the opposite end from SPF — no
+   cross-request memory, every segment starts at `uris[0]`. Dead simple, no
+   flapping logic, but no stickiness either. Worth knowing as the minimal
+   baseline; SPF's sticky `cdnPriority` + cooldown is deliberately more stateful.
+
+## Where SPF is distinctive
+
+- **CDN identity from URL origin, not a manifest signal.** Everyone else keys
+  on an explicit manifest token (`serviceLocation` for DASH, `PATHWAY-ID` for
+  HLS). SPF derives identity from the URL because Mux's `?redundant_streams=true`
+  doesn't emit pathway tags — and `getCdnId` makes it configurable (origin vs
+  `cdn=` param). A gap-filler the spec-bound players don't need but also can't
+  do; the right call for untagged redundancy.
+- **Failover as a pure rule in an existing selection chain**, not a dedicated
+  controller with its own state. Every other player carries a distinct object
+  (`ContentSteeringController`, `BaseURLSelector`, `CdnPrioritizer`,
+  `BaseUrlExclusionList`). SPF's failover is a constraint over tracks plus a
+  per-source cooldown timer (`setupFailoverMonitor`) — a smaller surface than
+  anyone else.
+- **Trip-on-first-failure with cooldown-only backoff** matches dash.js and the
+  spirit of media3, but is more aggressive than hls.js / rx-player, which both
+  count errors before tripping. SPF's bet — "absorbing transient blips is the
+  retry layer's job, once it exists" — is reasonable but means it is more
+  sensitive to a single blip than hls.js until network-resilience lands.
+
+## Sources
+
+Surveyed repositories (paths are within each repo, read against its default
+branch at survey time — verify before quoting as current):
+
+- video.js v8 http-streaming (VHS) — https://github.com/videojs/http-streaming — `src/`
+- hls.js — https://github.com/video-dev/hls.js — `src/controller/`
+- dash.js — https://github.com/Dash-Industry-Forum/dash.js — `src/streaming/`, `src/dash/controllers/`
+- shaka-player — https://github.com/shaka-project/shaka-player — `lib/net/`, `lib/util/content_steering_manager.js`
+- rx-player — https://github.com/canalplus/rx-player — `src/core/fetchers/`
+- Media3 / ExoPlayer — https://github.com/androidx/media — `libraries/exoplayer*/`
+- VLC — https://github.com/videolan/vlc — `modules/demux/adaptive/`
+- OSMF — https://github.com/denivip/OSMF — `framework/OSMF/org/osmf/net/httpstreaming/`
+
+External references:
+
+- [Mux Video — `?redundant_streams=true`](https://www.mux.com/docs/guides/play-back-on-multiple-cdns)
+- [HLS Content Steering (draft-pantos-hls-rfc8216bis)](https://datatracker.ietf.org/doc/html/draft-pantos-hls-rfc8216bis)
+- [DASH-IF Content Steering](https://dashif.org/docs/DASH-IF-CTS-00XX-Content-Steering-Community-Review.pdf)