Implement https://w3c.github.io/webcodecs/#videoencoder-interface

This is debugging-only.

Depends on D195696

Depends on D195697

They will be reused for the encoding side.

Depends on D195698

Depends on D195699

Depends on D195700

https://w3c.github.io/mediacapture-record/#bitratemode, we're not implementing
this on MediaRecorder, but are needed this for Web Codecs.

Depends on D195701

Depends on D195702

Depends on D195703

Depends on D195704

It intentionally more or less maps 1:1 since my refactoring.

Depends on D195705

Depends on D195706

It's a lot of code but there is no complexity. Messages are refptrs on this side
because it simplifies the code, remove lots of moves and the messages are const
anyways so uniqueptr don't bring anything.

There are two methods for encoding, because they don't have the same
signature: the video side can take options. Because I didn't want to refactor
yet, the audio side passes in empty options for now.

I'll reflect the changes bug 1865376 in a further patch, but they are similarly
needed.

Depends on D195707

If an EncoderAgent is already present, it means this is a reconfigure, and not
e.g. the first configure call after a reset, or the first configure call ever.

This attempts to set new parameters on a PEM. This can fail, it depends on what
the PEM can do. If it succeed, all is good and this is a success.

If it fails, or if we know that the reconfigure can't succeed (codec change, or
e.g. change between hw and software encoder), the encoder is flushed, destroyed,
and a new one is created with the new params.

Depends on D195708

No complexity there, just lots of glue.

Depends on D195709

I don't think it's terribly useful but it's there to call into the PEM to check
support. We'll see if we keep it.

Depends on D195710

Very similar to the decode side, except simpler because flushing's semantic
match.

Depends on D195711

This simplifies the rest of the code, in remaining patches.

Depends on D195712

There is a bit of overlap now conceptually between latency and bitstream format,
that will be cleaned up eventually. Web Codecs requires that latency and
bitstream format to be decoupled.

Depends on D195713

Depends on D195714

Depends on D195715

Similarly, this will be decoupled in the future, but this allows running and
passing tests for now. We don't ship it anyways.

Depends on D195716

Depends on D195717

Useful, supported, and required by Web Codecs.

Depends on D195718

Depends on D195719

From https://w3c.github.io/webcodecs/#enumdef-videocolorprimaries

bt709
Color primaries used by BT.709 and sRGB, as described by [H.273] section 8.1 table 2 value 1.

Depends on D195720

Depends on D195721

Depends on D196108

The previous code was causing issues when the frame rate wasn't available.

I've opened https://bugzilla.mozilla.org/show_bug.cgi?id=1869560 to revisit
this if feel we need to.

Depends on D196109

This is very useful to understand in the log what config is output in what
encoder output callback, with respect to configure(...) calls.

Depends on D196209

This logs when new configuration is output, and also logs the timestamp, to be
able to better match encode calls and output callbacks when logging.

Depends on D196210

When we a regular configure call happens, either it's the first configure call
after a reset (or at all), or we have the guarantee that the previous decoder
has been flushed, and so there are no packets in flight.

When we allow the FFmpegVideoEncoder to reconfigure on the fly, we should keep
track of what config each output should use.

Additionally, it's important to destroy / recreate the new EncoderAgent after
the packets have been output, so that ordering is correct. For this, we can
simply dispatch to the current thread, so that the runnable is after all the
output runnables. This was triggered by the WPT
reconfiguring-encoder.https.any.worker.html?vp9_p0, that doesn't output any
packet for the 16 first encode message, and the outputs 16 packets when the
encoder is flushed while reconfiguring.

Depends on D196211

Per FFmepg API's it has to be allocated on the heap and freed manually.
MakeScopeExit instead of uniqueptr + free policy is nicer I think because we
need to capture the FFmpegLibWrapper and all that.

Depends on D196212

https://searchfox.org/mozilla-central/rev/da48f565f70a57ac28862090828fbaa7fd8556f6/dom/media/VideoUtils.cpp#294
is one example of this.

Depends on D196213

Depends on D196336

Depends on D196337

There is a bit of a guess there as well, but a lot less than before.

https://bugzilla.mozilla.org/show_bug.cgi?id=1869825 has been filed to
investigate this, but there's various bits to it, in gfx and media.

Depends on D196338

Depends on D196339

Depends on D196214

Depends on D196340

Depends on D196472

Depends on D196473

Depends on D196474

This is ffmpeg 60 only, we might have to do it ourselves for other version. A
simple queue will work, since for all codecs we intend to support, it's one
packet per video frame.

Depends on D196475

Depends on D196476

Depends on D196477

Depends on D196478

Depends on D196456

Depends on D196480

This would error out later anyways, but causes an assertion in ASAN build
because the integer isn't part of the enum.

Depends on D196871

Depends on D196872

AV_PIX_FMT_YUV420P is only defined for ffmpeg version >= 54. For
version < 54, it's PIX_FMT_YUV420P.

ffmpeg::FFMPEG_PIX_FMT_YUV420P is the type automatically mapping those
pixel types for us.

Depends on D197019

AV_PIX_FMT_YUV420P is only defined for ffmpeg version >= 54. For
version < 54, it's PIX_FMT_YUV420P.

ffmpeg::FFMPEG_PIX_FMT_YUV420P is the type automatically mapping those
pixel types for us.

Depends on D197020

mozilla::layers::Image is incomplete type since it's forward declared
in MediaData.h, unless ImageContainer.h is included.

Depends on D197021

Using a template type, ConfigType, in EncoderAgent::Configure requires a
template implementation. Instead of defining the implementation
somewehre, taking a EncoderConfig in EncoderAgent makes it easier to
add a EncoderTemplate implementation in the future, as long as the
ConfigTypeInternal comes with a ToEncoderConfig() method.

Depends on D197022

Depends on D197023

Latest try push for those following along - https://treeherder.mozilla.org/jobs?repo=try&revision=0abe1d8aa051ff9ba7472f7571e52d1fd6c14f68

Depends on D197024

Depends on D197095

GetExtraData, which is removed in D196209, must be added back to
generate the extra data for H264 AVCC format, so the VideoEncoder can
forward those info with the encoded data.

Depends on D197137

This patch disables the WebCodecs encoding support on non-linux
platforms and adds some checks in WPTs to ensure that the execution
occurs only when the encoding is supported on the platform.

• Add checkEncoderSupport-like code to
  video-encoder-content-hint.https.any.js
• Add checkEncoderSupport to video-encoder-flush.https.any.js
• Add checkEncoderSupport to video-encoder.https.any.js
  and update video-encoder.https.any.js.ini

Depends on D197138

• For H264 encoding, FFmpegVideoEncoder requires a dynamically loaded
  ffmpeg with version > 57. However, the ffmpeg verions is 57 in ubuntu
  18.04, so video encoder module returns an "unsupported" for h264
• The av1 encoding of FFmpegVideoEncoder relies on the in-tree libaom,
  but the libaom is out-of-date. It's vendered in 2018, and it's pretty
  slow, so it's likely av1 encoding will get timeout in some situations.

Depends on D197139

Run:

./mach clang-format --path \
  dom/media/platforms/ffmpeg/FFmpegVideoEncoder.cpp

Depends on D196872

This patch uses RootedDictionary around EncodedVideoChunkMetadata,
VideoDecoderConfig and VideoColorSpaceInit in OutputEncodedData to
work around the JS hazard (unrooted) issues.

https://hg.mozilla.org/mozilla-central/rev/3695862104e5
https://hg.mozilla.org/mozilla-central/rev/9430aed8659c
https://hg.mozilla.org/mozilla-central/rev/029f5470314e
https://hg.mozilla.org/mozilla-central/rev/e903ac081227
https://hg.mozilla.org/mozilla-central/rev/2d5bc871fb9e
https://hg.mozilla.org/mozilla-central/rev/700cb49e7c8a
https://hg.mozilla.org/mozilla-central/rev/b6d97ebd3dda
https://hg.mozilla.org/mozilla-central/rev/ee31e639b046
https://hg.mozilla.org/mozilla-central/rev/bc4f18f14bd3
https://hg.mozilla.org/mozilla-central/rev/e3395cf67e5c
https://hg.mozilla.org/mozilla-central/rev/00a94805df47
https://hg.mozilla.org/mozilla-central/rev/5a2cef0729cc
https://hg.mozilla.org/mozilla-central/rev/3ce5091256fd
https://hg.mozilla.org/mozilla-central/rev/a65dfea63fc7
https://hg.mozilla.org/mozilla-central/rev/e59a8dc5add5
https://hg.mozilla.org/mozilla-central/rev/da42eb3f9664
https://hg.mozilla.org/mozilla-central/rev/23a3b0995f32
https://hg.mozilla.org/mozilla-central/rev/d8501e23246c
https://hg.mozilla.org/mozilla-central/rev/272ecdefa3db
https://hg.mozilla.org/mozilla-central/rev/b32edc2659a6
https://hg.mozilla.org/mozilla-central/rev/88d92a0509cd
https://hg.mozilla.org/mozilla-central/rev/0595d51073c6
https://hg.mozilla.org/mozilla-central/rev/a78bb8799ef4
https://hg.mozilla.org/mozilla-central/rev/bcb45500ba22
https://hg.mozilla.org/mozilla-central/rev/b9234a7fb938
https://hg.mozilla.org/mozilla-central/rev/be191f9ac293
https://hg.mozilla.org/mozilla-central/rev/2e334c8a1160
https://hg.mozilla.org/mozilla-central/rev/2214654fd856
https://hg.mozilla.org/mozilla-central/rev/12768abca8ac
https://hg.mozilla.org/mozilla-central/rev/7cbc5704562d
https://hg.mozilla.org/mozilla-central/rev/c781f30e5605
https://hg.mozilla.org/mozilla-central/rev/d43f0c82f214
https://hg.mozilla.org/mozilla-central/rev/279a5522919a
https://hg.mozilla.org/mozilla-central/rev/ce3fa8ca07ec
https://hg.mozilla.org/mozilla-central/rev/b7d997c42167
https://hg.mozilla.org/mozilla-central/rev/26c23ac47959
https://hg.mozilla.org/mozilla-central/rev/f0ac27321cf8
https://hg.mozilla.org/mozilla-central/rev/ea4d0db67f89
https://hg.mozilla.org/mozilla-central/rev/e99c66838796
https://hg.mozilla.org/mozilla-central/rev/4d35297ae4bb
https://hg.mozilla.org/mozilla-central/rev/0e68a1cc5f26
https://hg.mozilla.org/mozilla-central/rev/7a47354c58c4
https://hg.mozilla.org/mozilla-central/rev/cc1b04d5f70e
https://hg.mozilla.org/mozilla-central/rev/05a066dcfb52
https://hg.mozilla.org/mozilla-central/rev/44fac5680126
https://hg.mozilla.org/mozilla-central/rev/d3c7d90b2150
https://hg.mozilla.org/mozilla-central/rev/f55b735f283a
https://hg.mozilla.org/mozilla-central/rev/53c348265bb8
https://hg.mozilla.org/mozilla-central/rev/51dc14d395c6
https://hg.mozilla.org/mozilla-central/rev/6ecbca91effe
https://hg.mozilla.org/mozilla-central/rev/8436bfea59b9
https://hg.mozilla.org/mozilla-central/rev/db7a0f8d337d
https://hg.mozilla.org/mozilla-central/rev/306ad5c722c8
https://hg.mozilla.org/mozilla-central/rev/98d2ecb3f476
https://hg.mozilla.org/mozilla-central/rev/b1577155be1c

This bug implemented WebCodecs' video encoder but mainly focused on Linux for interop 2023. Although most of the code would be the same for other platforms, the implementations on other platforms should be tracked. I've opened a meta bug tracking this in bug 1872733, and I am going to move some dependencies there.

FF123 docs for this can be tracked in https://github.com/mdn/content/issues/31913

Can you please clarify what this actually delivers?

1. What desktop/mobile platforms?
   • The title says "Implement VideoEncoder on Linux" so from that I would assume that this fully supports whatever the spec says for VideoEncoder on FF123, but only on Linux desktop - not Windows, macOS or Firefox for android. However when I look at the gating preference dom.media.webcodecs.enabled I see that this is enabled on nightly for all versions, not restricted at all.
2. Does this match the specification for VideoEncoder (and EncodedVideoChunk, VideoFrame, VideoColorSpace) or are there some known/obvious missing parts?
   • On quick scan it looks like MDN IDL matches the spec, so I guess I'm asking confirmation/things I might have missed.
   • VideoDecoder appears to be gated by same preference, but I assume does not yet need docs.
3. Are there any special things that need to be said about this in terms of compatibility/what is delivered?
   • What I mean is that this is that the docs indicate you can use this to encode specific frames to formats supported by browser encoders, but presumably FF only supports some formats. Do we need to say what they are, or is this something people should query when needed?

This delivers nothing in release, we'll handle the release notes, there seem to have been something going on.

Thanks Paul, it's not just release notes. Even though this is not released we would usually record support for the feature behind pref dom.media.webcodecs.enabled in the compatibility data too. Unless there is a reason not to. Hence the question about what platforms and anything that we don't expect to be supported by this. If you "actively don't want" any support in the compatibility data or experimental features page at this point then we'd replace dev-doc-needed flag with dev-doc-complete and move on.

It's a bit bumpy at the minute, it only really works on Linux desktop, but support for all platforms should come in the next few weeks.

Answering your questions now that I understand what this is about:

1. Linux desktop only right now. I can take care of updating the browser compat thing MDN uses, I often do it myself when I implement / ship new Web API. This is Nightly-only, and is slated to ride the train and reach release in a few cycles.
2. This matches the spec, and the decoder side also implements the spec
3. The only thing I can think of is that encoding video frames the h264 codec isn't going to work on 5-6 years old linux, such as ubuntu 18.04 LTS, but this will fixed before release

Thanks a lot for handling this, lmk here on in chat if you need more info!

Hi Paul,

Yes, please do update the compatibility data! When you do that, can you please cross link from your PR to https://github.com/mdn/content/issues/31913 for my tracking. I assume you are going to take care of that in the near future (i.e. while it is still behind the pref)?

I'll add entry to the MDN Experimental features page for this - from what you've said I don't think there is much else to do now.

Thanks a lot for handling this, lmk here on in chat if you need more info!

I certainly will! From what I can tell the docs look pretty complete so not much else for me to ask at this point.
But if you think there is something I should doing for this you can tag me here or on the MDN github topic with @hamishwillee.

OK I've added a note about this in MDN experimental features/release notes. The docs look broadly good and I don't propose to touch them.
This is waiting on compatibility to be provided by PaulA.
Work for me complete for now, so have marked this as "dev-doc-complete".

Hey! Let us know if you need any help with the compat data in BCD for this feature. It’s less urgent since it’s an initial release on Linux only and the wider support is coming soon but we always try to keep the compact data up to date with the Firefox releases.

FYI I've added a BCD showing current progress https://github.com/mdn/browser-compat-data/pull/22289