People are using videos a lot more often for things that used to be GIFs (finally!), but this bug makes things painful on Firefox. It would be great if we could raise the priority here. I see looping jank on Linux, Android, and Windows.

Because our resource is limited. Before we implement this, I would like to add some telemetry probes to see how many video are used in that, comparing with non-looping video playback.

Got reminded of this issue due to an e-mail. Here to report that the issue is no longer present (fixed). It must have happened quite some time ago, but my current version is 103.0.2.

FYI we're tracking the Meet issue here - Bug 1789881 - Google Meet background video experiences black frame or stutter at video loop point

Depends on D159125

Depends on D159126

Depends on D159127

Depends on D159128

If the video looping is not seamless, when playback reaches to the end,
MDSM would trigger a seek in order to get the new frame from the start
position. That would notify the media element that the status of the next
frame is not available now due to seeking (NEXT_FRAME_UNAVAILABLE_SEEKING)
and causes the media element dispatching waiting event.

Above situaion shouldn't happen when we're in the seamless looping. The
added test covers that situation.

That ready state change also causes the google meet issue [1], because
the spec only allows capturing an image from a media element if it's
ready state is at least in HAVE_CURRENT_DATA.

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1789881#c3

Depends on D159217

This behavior is optional [1] and currently this test only passes on
Edge [2].

The reason of becoming timeout is because seamless looping now won't
change the media element's ready state. When looping is seamless,
element's ready state will change below HAVE_CURRENT_DATA (like what I
describe in D159218), and when we have enough data (HAVE_FUTURE_DATA or
HAVE_ENOUGH_DATA) the element would dispatch playing which causes the
condition check failure [3].

As now we're using seamless looping, now playing event will be resent
which causes the timeout.

[1] https://html.spec.whatwg.org/multipage/media.html#ready-states:eligible-for-autoplay-2
[2] https://wpt.fyi/results/html/semantics/embedded-content/media-elements/ready-states/autoplay-hidden.optional.html?label=experimental&view=subtest
[3] https://searchfox.org/mozilla-central/rev/76ccfc801e6b736c844cde3fddeab7a748fc8515/testing/web-platform/tests/html/semantics/embedded-content/media-elements/ready-states/autoplay-hidden.optional.html#27

Depends on D159218

This test ensures that we can always capture correct video frame via
canvas.

The black frame issue was discoverd in bug 1789881, where the canvas
capture is racing with the media element state change [1], which results
in capturing a black frame.

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1789881#c3

Depends on D159333

As we need some time to ensure that they can work well, so these patches will be targeting on Fx108.

Backed out 8 changesets (Bug 1262276) for causing failures in MediaDecoderStateMachine.cpp CLOSED TREE

Push with failures: https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&collapsedPushes=842382&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel&revision=0e8e7d877d08faad3a804b66002f02d93aac43f4&selectedTaskRun=dQ47p3zVTYWJURr7Yd_KnQ.0

Log: https://treeherder.mozilla.org/logviewer?job_id=393432218&repo=autoland&lineNumber=5475

Backout: https://hg.mozilla.org/integration/autoland/rev/20834bcffa8ef65a1848a13239df5438dfe3b9f3

And also mda failures: https://treeherder.mozilla.org/logviewer?job_id=393432096&repo=autoland

We don't want to trigger skip-to-next-keyframe after reaching video
EOS. Because now we've reset the demuxer to 0, and are going to request
data from start.

If playback hasn't looped back, the media time would still be too large,
which makes the reader think the playback is way behind and performs
nonecessary skipping which causes stutter.

Eg. Media is 10s long, reaching EOS at 8s, requesting data at 9s.

Assume media's keyframe interval is 3s, which means keyframes will
appear on 0s, 3s, 6s and 9s.

If we uses current media time as a threshold, the reader sees the next
key frame is 3s but the threashold is 9s, which usually happens when
the decoding is too slow.

But that is not the case for us, we should by pass the
skip-to-next-keyframe logic until the media loops back.

Depends on D160159

For pretty short media, (eg. gizmo-short.mp4), after reaching audio EOS,
the next audio sampe would be put on waiting because we can't adjust its
timestamp yet.

If so, we should stop prerolling otherwise the playback would never stop
because we won't ask for more audio data when there is always a data on
waiting.

Depends on D160160

This patch includes different test cases which caused problems before
during the development, and they are all some basic behavior tests.

Depends on D160163

When leaving looping state to another state, media data stored in the
media queue are already adjusted. If the new state requests a new data
but doesn't adjust its timestamp, then the data in the media queue will
be out of order.

If that happens on video data, it would causes a/v unsync and the video
frame would be discarded because it doesn't catch up with the clock
time, which might have grown a lot via looping multiple times.

The example transitions from the looping state which would encounter
this situation are buffering state (decoding too slow), decoding state
(cancel looping) and video only seek (bkg video resume).

In the premise of letting the clock time keep growing, we would need to
put the offset to somewhere independent to states. Therefore, we choose
to let the media queue do the task of the timestamp adjustment.

So even if we leave the looping state, the new coming data would be
adjusted their timestamp correctly and match the clock time. If we enter
the looping state again, we can also smoothly keep adding more offset to
all future data.

Depends on D160163

When requesting data from demuxer, it should always use normalized data,
which is within [0, original-track-duration]. So we need to adjust seek
target time in order not to use clock time which has grown over the
duration due to multiple times of looping (that time is larger than
duration).

Depends on D160164

Using mozPresentedFrames is incorrect, because that represents how
many frames the video sink sends to the compositor, and that can include
the same frame multiple time. [1]

Due to the seamless looping, now we could have more frames in the video
queue, so mozPresentedFrames could be much larger than the actual
frames already be rendered. Therefore, we should use mozPaintedFrames
instead, which is closer to the situation we wanted to measure (submit
native layer to screen)

This patch also increases the playback rate and the threshold in the
test in order to reduce test running time.

[1] https://searchfox.org/mozilla-central/rev/fe5c9c39a879b07d5b629257f63d825c3c8cd0ed/dom/media/mediasink/VideoSink.cpp#515

Depends on D160699

Per offline discussion with Kesley, we decided to disable seamless
looping for these tests first, and then spend time to figure the gfx
side crash due to not be able to manage texture's lifetime correctly.

Depends on D160700

This doesn't work for seamless looping because the video end time can
be larger than the media time. We would need to revisit these mechaniam
later to see if we can have a better way to skip to next frame.

Depends on D160701

Depends on D160864

https://hg.mozilla.org/mozilla-central/rev/1df4774b2c53
https://hg.mozilla.org/mozilla-central/rev/3e647187e78b
https://hg.mozilla.org/mozilla-central/rev/f6c98cf1d45f
https://hg.mozilla.org/mozilla-central/rev/6672431765dc
https://hg.mozilla.org/mozilla-central/rev/1dd4093cd374
https://hg.mozilla.org/mozilla-central/rev/55e490e2de4c
https://hg.mozilla.org/mozilla-central/rev/1dac017b6089
https://hg.mozilla.org/mozilla-central/rev/e807704e1c9b
https://hg.mozilla.org/mozilla-central/rev/d122b10f03ad
https://hg.mozilla.org/mozilla-central/rev/abd21243cfda
https://hg.mozilla.org/mozilla-central/rev/44e50217fdc9
https://hg.mozilla.org/mozilla-central/rev/e43f99440e29
https://hg.mozilla.org/mozilla-central/rev/12aaeb776da1
https://hg.mozilla.org/mozilla-central/rev/9e9f7deff834
https://hg.mozilla.org/mozilla-central/rev/b1a5972d0cee
https://hg.mozilla.org/mozilla-central/rev/ee73611a2993
https://hg.mozilla.org/mozilla-central/rev/f436cf1a01cc
https://hg.mozilla.org/mozilla-central/rev/1d0656c6185d