Skip to main content

WebCodecs

warning

Unstable API: This package is experimental. We might change the API at any time, until we remove this notice.

parseMedia() is able to extract tracks and frames in a format that is suitable for passing to WebCodecs APIs.

Support​

We plan support for extracting frames from common containers and codecs.
This is the current support:

From MP4 videos:

  • h.264
  • h.265
  • AV1
  • AAC
  • MP3
  • PCM

From WebM videos:

  • h.264
  • h.265
  • vp8
  • vp9
  • AV1
  • AAC
  • MP3
  • Opus
  • PCM
  • Vorbis

API​

Reading video and audio frames
tsx
import {parseMedia} from '@remotion/media-parser';
 
const result = await parseMedia({
  src: 'https://remotion.dev/bbb.mp4',
  fields: {},
  onVideoTrack: (track) => {
    // const decoder = new VideoDecoder({...})
    // decoder.configure({
    //   codec: track.codec,
    //   codedWidth: track.width,
    //   codedHeight: track.height
    // });
    return (sample) => {
      // const frame = new EncodedVideoChunk({
      //   type: sample.type,
      //   timestamp: sample.timestamp,
      //   duration: sample.duration,
      //   data: sample.bytes,
      // })
      // decoder.decode(frame);
      console.log('New video sample', sample);
    }
  },
  onAudioTrack: (track) => {
    // const decoder = new AudioDecoder({...})
    return (sample) => {
      // const frame = new EncodedAudioChunk({
      // });
      console.log('New audio sample', sample);
    }
  }
});
Reading video and audio frames
tsx
import {parseMedia} from '@remotion/media-parser';
 
const result = await parseMedia({
  src: 'https://remotion.dev/bbb.mp4',
  fields: {},
  onVideoTrack: (track) => {
    // const decoder = new VideoDecoder({...})
    // decoder.configure({
    //   codec: track.codec,
    //   codedWidth: track.width,
    //   codedHeight: track.height
    // });
    return (sample) => {
      // const frame = new EncodedVideoChunk({
      //   type: sample.type,
      //   timestamp: sample.timestamp,
      //   duration: sample.duration,
      //   data: sample.bytes,
      // })
      // decoder.decode(frame);
      console.log('New video sample', sample);
    }
  },
  onAudioTrack: (track) => {
    // const decoder = new AudioDecoder({...})
    return (sample) => {
      // const frame = new EncodedAudioChunk({
      // });
      console.log('New audio sample', sample);
    }
  }
});

Why is this useful?​

To use WebCodecs, the binary format of a video needs to be understood and parsed.
Parsing this is a meticulous and non-trivial task.
parseMedia() allows to to read an arbitrary media file and interface with it regardless of container, video codec and audio codec.

How can I use this?​

You can extract frames by passing the track data into a VideoDecoder.

Retrieved sample values can be passed to the EncodedVideoChunk() constructor and then passed to VideoDecoder.decode().

To encode the video into a different format is a non-trivial task still.
We plan to offer additional APIs in the future to help with this.