mirror of
https://github.com/zoriya/v10.git
synced 2026-08-16 02:45:09 +00:00
227 lines
10 KiB
Markdown
227 lines
10 KiB
Markdown
# RFC: Unified Media API
|
||
|
||
## Summary
|
||
|
||
This RFC proposes a **unified media API** that extends the [HTMLMediaElement](https://html.spec.whatwg.org/multipage/media.html#htmlmediaelement) to support different media types (video, HLS, DASH, YouTube, etc.) in a consistent way. The API is designed to:
|
||
|
||
- **Extend** the existing HTMLMediaElement surface rather than replace it. Implementations may expose the **full** HTMLMediaElement surface or a **subset** of it; the unified contract is "HTMLMediaElement-like" where the subset is sufficient for the integration (e.g. play/pause, currentTime, seekable).
|
||
- **Support multiple integration points:** for HTML, web components that implement this API; for React and other frameworks, an API that extends `EventTarget` and exposes the same surface (full or subset).
|
||
- **Add custom extensions** for behavior not covered by HTMLMediaElement, specifically:
|
||
- **Stream type** — distinguish live vs. on-demand content.
|
||
- **Live edge** — model the “live edge” of live streams (window, seekable end, seek-to-live).
|
||
- **Renditions list** — discover and select video/audio quality levels (renditions) for HTTP Adaptive Streaming (HAS).
|
||
|
||
## Motivation
|
||
|
||
## Unified control across media types
|
||
|
||
Players today deal with many sources: native `<video>`, HLS, DASH, YouTube, etc. Each often has its own API and event model. A unified media API allows:
|
||
|
||
- One control surface for play/pause, seeking, volume, and **stream type**, **live edge**, and **renditions**, regardless of backend.
|
||
- Consistent behavior for web components and frameworks (e.g. React) that sit on top of a single abstraction.
|
||
|
||
### Gaps in the platform
|
||
|
||
Current HTMLMediaElement does not address:
|
||
|
||
1. **Stream type** — UIs need to know if content is live or on-demand (e.g. show/hide seek bar, live badge, DVR controls). Today this is inferred from `duration` or format-specific logic, which is brittle (e.g. ended live vs. VOD).
|
||
2. **Live edge** — For live/HAS, “live” is a window, not a single time. UIs need a clear notion of “at live” and “seek to live” that respects HOLD-BACK, target duration, and segment boundaries.
|
||
3. **Renditions** — Users and UIs need to see available quality levels and select a rendition; today this is player-specific and not standardized.
|
||
|
||
By extending the media element (or an EventTarget-based equivalent) with **stream type**, **live edge**, and **renditions list** APIs, we give UIs a single, predictable way to implement live indicators, seek-to-live, quality selectors, and stream-type–aware chrome.
|
||
|
||
## Design principles
|
||
|
||
- **Extension of HTMLMediaElement** — New behavior is added via partial interfaces and constraints on existing attributes (e.g. `seekable`), not by replacing the element.
|
||
- **Subset of HTMLMediaElement allowed** — A conforming implementation may expose the full HTMLMediaElement surface or a **subset** of it. The contract is "HTMLMediaElement-like": consumers can rely on the subset that is documented as required for a given use case (e.g. `play`, `pause`, `currentTime`, `seekable` for basic playback). Backends that cannot support certain attributes (e.g. `audioTracks` for some HAS) may omit them or expose them as no-ops/empty where specified.
|
||
- **Web components and frameworks** — For HTML, the API can be implemented by custom elements that wrap or emulate a media element. For React and others, the same contract can be implemented by an object that extends `EventTarget` and exposes the same properties and events (full or subset).
|
||
- **Custom extensions only where needed** — Use native media APIs where they suffice; add **stream type**, **live edge**, and **renditions list** only where the platform does not already define the behavior.
|
||
|
||
## API overview
|
||
|
||
The unified media API adds the following to a media element (or EventTarget-based equivalent). The base surface is **HTMLMediaElement** (or a subset thereof); implementations may expose the full HTMLMediaElement API or only the subset needed for their backend and use case.
|
||
|
||
| Area | Addition | Purpose |
|
||
| ----------- | --------------------------------------------- | ------------------------------------------------------------------------------- |
|
||
| Stream type | `streamType`, `streamtypechange` | Know if content is live or on-demand and adapt UI (seek bar, live badge, etc.). |
|
||
| Live edge | `liveEdgeStart`, constrained `seekable.end()` | “At live” detection and “seek to live” for live/HAS. |
|
||
| Renditions | `videoRenditions`, `audioRenditions` | List and select quality levels for HAS. |
|
||
|
||
- **HTMLMediaElement subset** — A conforming implementation may implement only a subset of HTMLMediaElement (e.g. core playback: `play`, `pause`, `currentTime`, `duration`, `seekable`, `paused`, `ended`). The type system and documentation will define which subset is required for which integration.
|
||
- **Custom extensions** — Implementations may expose the custom extensions only when applicable (e.g. `liveEdgeStart` for live; renditions for HAS).
|
||
|
||
```ts
|
||
declare const MediaEvents = [
|
||
'abort',
|
||
'canplay',
|
||
'canplaythrough',
|
||
'durationchange',
|
||
'emptied',
|
||
'encrypted',
|
||
'ended',
|
||
'error',
|
||
'loadeddata',
|
||
'loadedmetadata',
|
||
'loadstart',
|
||
'pause',
|
||
'play',
|
||
'playing',
|
||
'progress',
|
||
'ratechange',
|
||
'seeked',
|
||
'seeking',
|
||
'stalled',
|
||
'suspend',
|
||
'timeupdate',
|
||
'volumechange',
|
||
'waiting',
|
||
'waitingforkey',
|
||
'resize',
|
||
'enterpictureinpicture',
|
||
'leavepictureinpicture',
|
||
'webkitbeginfullscreen',
|
||
'webkitendfullscreen',
|
||
'webkitpresentationmodechanged',
|
||
] as const;
|
||
|
||
interface MediaBaseApi {
|
||
play(): Promise<void>;
|
||
paused: boolean;
|
||
}
|
||
|
||
interface MediaFullApi {
|
||
// Properties
|
||
src: string;
|
||
srcObject: MediaStream | MediaSource | Blob | null;
|
||
currentSrc: string; // read only
|
||
crossOrigin: string | null;
|
||
preload: 'none' | 'metadata' | 'auto';
|
||
autoplay: boolean;
|
||
loop: boolean;
|
||
controls: boolean;
|
||
controlsList: DOMTokenList; // read only
|
||
volume: number; // 0.0 to 1.0
|
||
muted: boolean;
|
||
defaultMuted: boolean;
|
||
currentTime: number;
|
||
defaultPlaybackRate: number;
|
||
playbackRate: number;
|
||
duration: number; // read only, NaN if unknown, Infinity for live
|
||
ended: boolean; // read only
|
||
seeking: boolean; // read only
|
||
seekable: TimeRanges; // read only
|
||
buffered: TimeRanges; // read only
|
||
played: TimeRanges; // read only
|
||
networkState: number; // read only (NETWORK_EMPTY, LOADING, LOADED, NO_SOURCE)
|
||
readyState: number; // read only (HAVE_NOTHING, HAVE_METADATA, HAVE_CURRENT_DATA, HAVE_FUTURE_DATA, HAVE_ENOUGH_DATA)
|
||
error: MediaError | null; // read only
|
||
disableRemotePlayback: boolean;
|
||
remote: RemotePlayback; // read only
|
||
mediaKeys: MediaKeys | null; // read only, secure context
|
||
audioTracks: AudioTrackList; // read only
|
||
videoTracks: VideoTrackList; // read only
|
||
textTracks: TextTrackList; // read only
|
||
|
||
// Custom: requires polyfill in some browsers
|
||
addVideoTrack(track: VideoTrack): void;
|
||
addAudioTrack(track: AudioTrack): void;
|
||
removeVideoTrack(track: VideoTrack): void;
|
||
removeAudioTrack(track: AudioTrack): void;
|
||
|
||
// Methods
|
||
pause(): void;
|
||
load(): void;
|
||
canPlayType(type: string): 'probably' | 'maybe' | '';
|
||
addTextTrack(kind: TextTrackKind, label?: string, language?: string): TextTrack;
|
||
setMediaKeys(mediaKeys: MediaKeys | null): Promise<void>; // secure context
|
||
}
|
||
|
||
export interface RenditionsCapability {
|
||
videoRenditions: VideoRenditionList;
|
||
audioRenditions: AudioRenditionList;
|
||
}
|
||
|
||
export interface StreamTypeCapability {
|
||
streamType: 'live' | 'on-demand';
|
||
}
|
||
|
||
export interface LiveEdgeCapability {
|
||
liveEdgeStart: number;
|
||
}
|
||
|
||
interface MediaApi extends MediaBaseApi,
|
||
Partial<MediaFullApi>,
|
||
Partial<RenditionsCapability>,
|
||
Partial<StreamTypeCapability>,
|
||
Partial<LiveEdgeCapability>
|
||
{
|
||
// ...
|
||
}
|
||
|
||
export interface Delegate {
|
||
attach?(target: EventTarget): void;
|
||
detach?(): void;
|
||
}
|
||
|
||
// DelegateMixin intercepts get/set/call to route through a Delegate before falling through to the base class.
|
||
export function DelegateMixin<Base extends Constructor, D extends Constructor<Delegate>>(
|
||
BaseClass: Base,
|
||
DelegateClass: D
|
||
): Constructor<InstanceType<Base> & InstanceType<D>>;
|
||
|
||
|
||
// ProxyMixin creates a class that proxies methods and properties to the attached target.
|
||
export const ProxyMixin = <T extends EventTarget>(
|
||
PrimaryClass: AnyConstructor<T>,
|
||
...AdditionalClasses: AnyConstructor<EventTarget>[]
|
||
) => class MediaProxy {
|
||
target: EventTarget | null;
|
||
attach(target: EventTarget): void;
|
||
detach(): void;
|
||
|
||
get(prop: keyof EventTarget): any;
|
||
set(prop: keyof EventTarget, val: any): void;
|
||
call(prop: keyof EventTarget, ...args: any[]): any;
|
||
}
|
||
|
||
// Pre-built proxy for video elements.
|
||
export const VideoProxy = ProxyMixin(HTMLVideoElement, HTMLMediaElement, EventTarget);
|
||
|
||
// CustomVideoElement wraps a native <video> in a custom element.
|
||
export const CustomVideoElement = CustomMediaMixin(HTMLElement, { tag: 'video' });
|
||
export const CustomAudioElement = CustomMediaMixin(HTMLElement, { tag: 'audio' });
|
||
```
|
||
|
||
### Example: HLS delegate
|
||
|
||
A delegate intercepts property access for engine-specific behavior (e.g. `src`), while the base class (CustomVideoElement or VideoProxy) handles standard HTMLMediaElement properties.
|
||
|
||
```ts
|
||
import Hls from 'hls.js';
|
||
import { type Delegate, DelegateMixin } from '../core/media/delegate';
|
||
import { CustomVideoElement } from './custom-media-element';
|
||
import { VideoProxy } from './proxy';
|
||
|
||
class HlsMediaDelegateBase implements Delegate {
|
||
#engine = Hls.isSupported() ? new Hls() : null;
|
||
|
||
get engine(): Hls | null { return this.#engine; }
|
||
|
||
attach(target: EventTarget): void {
|
||
this.#engine?.attachMedia(target as HTMLMediaElement);
|
||
}
|
||
|
||
detach(): void { this.#engine?.detachMedia(); }
|
||
destroy(): void { this.#engine?.destroy(); }
|
||
|
||
set src(src: string) { this.#engine?.loadSource(src); }
|
||
get src(): string { return this.#engine?.url ?? ''; }
|
||
}
|
||
|
||
// Web component — extends HTMLElement via CustomVideoElement.
|
||
export class HlsCustomMedia extends DelegateMixin(CustomVideoElement, HlsMediaDelegateBase) {}
|
||
|
||
// React / framework — extends EventTarget via VideoProxy.
|
||
export class HlsMedia extends DelegateMixin(VideoProxy, HlsMediaDelegateBase) {}
|
||
```
|