Closed Bug 1550633 Opened 5 years ago Closed 5 years ago

[webvtt] HTMLTrackElement should not load resource when its text track is disable

Categories

(Core :: Audio/Video: Playback, defect, P2)

defect

Tracking

()

RESOLVED FIXED
mozilla69
Tracking Status
firefox69 --- fixed

People

(Reporter: alwu, Assigned: alwu, NeedInfo)

References

(Blocks 2 open bugs)

Details

(Keywords: dev-doc-complete)

Attachments

(23 files, 1 obsolete file)

47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review

According to the spec step2 [1], we should not load resource for the track element which text track is disabled.

[1] https://html.spec.whatwg.org/multipage/media.html#start-the-track-processing-model

According to the spec step2 [1], we should not load resource for the track element which text track is disabled.

[1] https://html.spec.whatwg.org/multipage/media.html#start-the-track-processing-model

According to the spec [1], the user agent must start the track processing model when text track has its text track mode changed.

[1] https://html.spec.whatwg.org/multipage/media.html#sourcing-out-of-band-text-tracks:start-the-track-processing-model

Showing track's kind in debug log is helpful.

According to spec [1], it doesn't mention that we have to run track selection in stable state, it just said "the user agent must queue a task to run the following steps".

[1] https://html.spec.whatwg.org/multipage/media.html#honor-user-preferences-for-automatic-text-track-selection

In honor user preferences for automatic text track selection [1], we would set did-perform-automatic-track-selection flag to true [2], and then we won't execute automatic track selection anymore [3].

It means that we would only do automatic track selection one time, and then user has to enable newly added track explicitly by changing its mode.

In this test, we have done the automatic track selection when we added the metadata track to media element's text track list, so we won't run it again, even if the newly added track has default attribute.

Therefore, we have to enable the caption track explicitly. In addition, add the missing event parameter for the function trackAdded().

[1] https://html.spec.whatwg.org/multipage/media.html#honor-user-preferences-for-automatic-text-track-selection
[2] https://html.spec.whatwg.org/multipage/media.html#sourcing-out-of-band-text-tracks:did-perform-automatic-track-selection-2
[3] https://html.spec.whatwg.org/multipage/media.html#sourcing-out-of-band-text-tracks:did-perform-automatic-track-selection

According to the spec [1], we should empty track's cue list whenever a track element has its src attribute set, changed, or removed.

[1] https://html.spec.whatwg.org/multipage/media.html#sourcing-out-of-band-text-tracks:attr-track-src

Failed to load [1] indicates that the text track was enabled, but when the user agent attempted to obtain it, this failed in some way.

According to the spec [2], if fetching fails for any reason (network error, the server returns an error code, CORS fails, etc), or if URL is the empty string, then queue a task to first change the text track readiness state to failed to load and then fire an event named error at the track element.

In addition, when we remove src attribute from a loading or loaded track element, we should also set it to failed to load.

[1] https://html.spec.whatwg.org/multipage/media.html#text-track-readiness-state
[2] https://html.spec.whatwg.org/multipage/media.html#sourcing-out-of-band-text-tracks:text-track-failed-to-load

As the src url is the same, we have no need to reload the resource.

There are too many self used in the lambda, we can just capture this to remove redudant self.

To reduce non-neccesary input, we can always capture this and print the address in the log, and use same log level for all of them.

According to the spec [1], the text track list of cues is initially empty. It is dynamically modified when the referenced file is parsed.

Therefore, after reset the track element's url, the number of cues of the text track should be zero, until we start parsing resource and add cue into the list.

[1]
https://html.spec.whatwg.org/multipage/media.html#sourcing-out-of-band-text-tracks:text-track-list-of-cues
https://html.spec.whatwg.org/multipage/media.html#sourcing-out-of-band-text-tracks:text-track-list-of-cues-3

As now we won't start loading for disable tracks, so we have to enable the track by changing its track mode explicitly or adding default attribute before the media element running the automatic track selection.

There are some common changes for these tests.

(1) listen track element's load event
There are some tests listening video's loadedmetadata, but it's wrong. The loading process of media element and track element are completely non-related.
If you would like to check track element's status, you should wait for track element's load event.

(2) enable track explictly
If the text track which has default attribute is added to the media element before the media element starts running automatic track selection [1], then it would be enable by the media element.
Otherwise, you have to enable track explicitly by changing its track mode.

(3) refactor tests
Refactor those tests' structure in order to make them more readable, and add the comment to show what the test purpose is for each test.

[1] https://html.spec.whatwg.org/multipage/media.html#sourcing-out-of-band-text-tracks:text-track-7

Attachment #9065276 - Attachment description: Bug 1550633 - part2 : check whether we have to load resource when the track's mode changes. → Bug 1550633 - part2 : maybe start loading resource when the track's mode changes.
Attachment #9065278 - Attachment description: Bug 1550633 - part4 : no need to run text selection in stable state. → Bug 1550633 - part4 : no need to run text track selection in stable state.

These elemenets are useless and we can run tests without them.

These tests didn't use region at all, so we have no need to set the pref.

Create test elements in HTML beforehead, which can remove unnecessary JS codes and make test cleaner.

Use named function for callback to reduce the indentation.

This patch do two things in order to trigger loading for track element and wait for correct event to check track's and cues' status after loading finished.

(1) listen track element's load event
There are some tests listening video's loadedmetadata, but it's wrong. The loading process of media element and track element are completely non-related.
If you would like to check track element's status, you should wait for track element's load event.

(2) enable track explictly
If the text track which has default attribute is added to the media element before the media element starts running automatic track selection [1], then it would be enable by the media element.
Otherwise, you have to enable track explicitly by changing its track mode.

[1] https://html.spec.whatwg.org/multipage/media.html#sourcing-out-of-band-text-tracks:text-track-7

Refactor those tests' structure in order to make them more readable, and add the comment to show what the test purpose is for each test.

Attachment #9065803 - Attachment is obsolete: true

According to spec [1], if the track URL changes so that it is no longer equal to URL while fetching is ongoing, we need to change the text track readiness state tofailed to load and dispatch error.

It didn't mention we have to dispatch error if track element has finished loading.

[1] https://html.spec.whatwg.org/multipage/media.html#sourcing-out-of-band-text-tracks:text-track-failed-to-load-3

This test is used to ensure that we queue 'honor user preferences for automatic text track selection' as a marco task, not a mirco task.

In this test, we would trigger a media event before queuing a text track selection task, and check the text track's mode to know whether the text track selection runs after the task for media event.

The channel might not be created correctly if we pass invaild url (eg. "invalid://url"), we should handle this error.

Sometime wpt runs test even before the document becomes visible, which would delay video.play() and cause play() running in wrong order.

Attachment #9066579 - Attachment description: Bug 1550633 - part16 : turn off 'media.block-autoplay-until-in-foreground' in wpt. → Bug 1550633 - part16 : turn off the pref 'media.block-autoplay-until-in-foreground' in wpt.
Pushed by alwu@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/12fc5abcd066
part1 : HTMLTrackElement should not load resource when its text track is disable. r=jya
https://hg.mozilla.org/integration/autoland/rev/81545b2c0b58
part2 : maybe start loading resource when the track's mode changes. r=jya
https://hg.mozilla.org/integration/autoland/rev/4c8fd0720cc2
part3 : add debug log to show track's kind. r=jya
https://hg.mozilla.org/integration/autoland/rev/4346465d73b1
part4 : no need to run text track selection in stable state. r=bzbarsky
https://hg.mozilla.org/integration/autoland/rev/f75c225dc273
part5 : modify and enable test 'track-mode-not-changed-by-new-track.html'. r=jya
https://hg.mozilla.org/integration/autoland/rev/679ce0bdbf18
part6 : clear track's cues list whenever track element's src attribute set, changed or removed. r=jya,baku
https://hg.mozilla.org/integration/autoland/rev/d8fdfd9505bf
part7 : change ready state to 'FAILED_TO_LOAD' and dispatch 'error' event when we have no 'src' or have an empty url. r=jya
https://hg.mozilla.org/integration/autoland/rev/9ad7d7b5ef6d
part8 : no need to reload if the new url is the same. r=jya
https://hg.mozilla.org/integration/autoland/rev/e4128df1406b
part9 : capture 'this' in lambda. r=jya
https://hg.mozilla.org/integration/autoland/rev/fefc8d667b9c
part10 : automatically capture and print the address of track element in debug log and add new log. r=jya
https://hg.mozilla.org/integration/autoland/rev/4b193e156faa
part11 : modify wpt 'track-element-src-change.html' and 'track-element-src-change-error.html'. r=jya
https://hg.mozilla.org/integration/autoland/rev/ed0f76cca0d9
part11.1 - modify 'track-element-src-change-error.html'. r=jya
https://hg.mozilla.org/integration/autoland/rev/abc099f50543
part12 : enable wpts. r=jya
https://hg.mozilla.org/integration/autoland/rev/d338d667dcbd
part13.1 - remove unnecessary elements. r=jya
https://hg.mozilla.org/integration/autoland/rev/9a5e3e6b8c21
part13.2 - no need to set pref "media.webvtt.regions.enabled". r=jya
https://hg.mozilla.org/integration/autoland/rev/27e03b195718
part13.3 - create elements in HTML, not in JS. r=jya
https://hg.mozilla.org/integration/autoland/rev/22213d6f047c
part13.4 : don't use anonymous function. r=jya
https://hg.mozilla.org/integration/autoland/rev/e22ef6f848aa
part13.5 - wait text track element's 'load' event. r=jya
https://hg.mozilla.org/integration/autoland/rev/97f61668f78c
part13.6 - refactor tests. r=jya
https://hg.mozilla.org/integration/autoland/rev/bf1426571565
part14 : add a test for 'honor user preferences for automatic text track selection'. r=bzbarsky
https://hg.mozilla.org/integration/autoland/rev/a42883570179
part15 : return error when we failed to create channel. r=jya
https://hg.mozilla.org/integration/autoland/rev/87067f045e1e
part16 : turn off the pref 'media.block-autoplay-until-in-foreground' in wpt. r=jgraham
Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/16965 for changes under testing/web-platform/tests
Upstream web-platform-tests status checks passed, PR will merge once commit reaches central.
Upstream PR was closed without merging

As now we won't automatically load disabled text track, we have to mark track as default in order to trigger loading.

Flags: needinfo?(alwu)
Pushed by alwu@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/9893ccf1f8ad
part1 : HTMLTrackElement should not load resource when its text track is disable. r=jya
https://hg.mozilla.org/integration/autoland/rev/be178934cbbb
part2 : maybe start loading resource when the track's mode changes. r=jya
https://hg.mozilla.org/integration/autoland/rev/b15cc6e689e7
part3 : add debug log to show track's kind. r=jya
https://hg.mozilla.org/integration/autoland/rev/def294ad0efb
part4 : no need to run text track selection in stable state. r=bzbarsky
https://hg.mozilla.org/integration/autoland/rev/26c3c521ce27
part5 : modify and enable test 'track-mode-not-changed-by-new-track.html'. r=jya
https://hg.mozilla.org/integration/autoland/rev/224eadc5ef27
part6 : clear track's cues list whenever track element's src attribute set, changed or removed. r=jya,baku
https://hg.mozilla.org/integration/autoland/rev/8024fcefe8b1
part7 : change ready state to 'FAILED_TO_LOAD' and dispatch 'error' event when we have no 'src' or have an empty url. r=jya
https://hg.mozilla.org/integration/autoland/rev/470d8fae0461
part8 : no need to reload if the new url is the same. r=jya
https://hg.mozilla.org/integration/autoland/rev/afaef937598b
part9 : capture 'this' in lambda. r=jya
https://hg.mozilla.org/integration/autoland/rev/b6e8370b6efb
part10 : automatically capture and print the address of track element in debug log and add new log. r=jya
https://hg.mozilla.org/integration/autoland/rev/d85485f76233
part11 : modify wpt 'track-element-src-change.html' and 'track-element-src-change-error.html'. r=jya
https://hg.mozilla.org/integration/autoland/rev/2aaaa42c409b
part11.1 - modify 'track-element-src-change-error.html'. r=jya
https://hg.mozilla.org/integration/autoland/rev/4265dd65cec3
part12 : enable wpts. r=jya
https://hg.mozilla.org/integration/autoland/rev/e9dea69efffe
part13.1 - remove unnecessary elements. r=jya
https://hg.mozilla.org/integration/autoland/rev/2d4773f2dc6c
part13.2 - no need to set pref "media.webvtt.regions.enabled". r=jya
https://hg.mozilla.org/integration/autoland/rev/d4a6c1337f89
part13.3 - create elements in HTML, not in JS. r=jya
https://hg.mozilla.org/integration/autoland/rev/fd1afdd721e3
part13.4 : don't use anonymous function. r=jya
https://hg.mozilla.org/integration/autoland/rev/0f6ddd898e89
part13.5 - wait text track element's 'load' event. r=jya
https://hg.mozilla.org/integration/autoland/rev/29487a59512d
part13.6 - refactor tests. r=jya
https://hg.mozilla.org/integration/autoland/rev/583d4e0945ad
part14 : add a test for 'honor user preferences for automatic text track selection'. r=bzbarsky
https://hg.mozilla.org/integration/autoland/rev/bbd8f62b9bff
part15 : return error when we failed to create channel. r=jya
https://hg.mozilla.org/integration/autoland/rev/e1a33576d6ad
part16 : turn off the pref 'media.block-autoplay-until-in-foreground' in wpt. r=jgraham
https://hg.mozilla.org/integration/autoland/rev/ce4a146da49d
part17 : modify test 'browser_cache.js'. r=timhuang
Upstream web-platform-tests status checks passed, PR will merge once commit reaches central.
Upstream PR was closed without merging

I've already updated the fix in patch17. In order to ensure we won't break any test again, pushing patches to try server before relanding all of them.
https://treeherder.mozilla.org/#/jobs?repo=try&revision=825ad84c20cb0b10b247c6c501e6d6c704420e73

Flags: needinfo?(alwu)
Attachment #9067118 - Attachment description: Bug 1550633 - part17 : modify test 'browser_cache.js'. → Bug 1550633 - part17 : add 'default' for track elements in 'browser_cache.js ' and 'trackerFrame.html '.
Attachment #9067118 - Attachment description: Bug 1550633 - part17 : add 'default' for track elements in 'browser_cache.js ' and 'trackerFrame.html '. → Bug 1550633 - part17 : enable loading for track elements in 'browser_cache.js ' and 'trackerFrame.html '.
Pushed by alwu@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/572e5b0bcab9
part1 : HTMLTrackElement should not load resource when its text track is disable. r=jya
https://hg.mozilla.org/integration/autoland/rev/6d16370ca0cf
part2 : maybe start loading resource when the track's mode changes. r=jya
https://hg.mozilla.org/integration/autoland/rev/a5ed9d3975c4
part3 : add debug log to show track's kind. r=jya
https://hg.mozilla.org/integration/autoland/rev/c90139d2c836
part4 : no need to run text track selection in stable state. r=bzbarsky
https://hg.mozilla.org/integration/autoland/rev/7a20deaf11bc
part5 : modify and enable test 'track-mode-not-changed-by-new-track.html'. r=jya
https://hg.mozilla.org/integration/autoland/rev/b19effb108a9
part6 : clear track's cues list whenever track element's src attribute set, changed or removed. r=jya,baku
https://hg.mozilla.org/integration/autoland/rev/01efc1080b51
part7 : change ready state to 'FAILED_TO_LOAD' and dispatch 'error' event when we have no 'src' or have an empty url. r=jya
https://hg.mozilla.org/integration/autoland/rev/bb4756110ece
part8 : no need to reload if the new url is the same. r=jya
https://hg.mozilla.org/integration/autoland/rev/079555c7d41b
part9 : capture 'this' in lambda. r=jya
https://hg.mozilla.org/integration/autoland/rev/6c49cf7caac3
part10 : automatically capture and print the address of track element in debug log and add new log. r=jya
https://hg.mozilla.org/integration/autoland/rev/4d15400f8a6d
part11 : modify wpt 'track-element-src-change.html' and 'track-element-src-change-error.html'. r=jya
https://hg.mozilla.org/integration/autoland/rev/f4d1f1b6e2c3
part11.1 - modify 'track-element-src-change-error.html'. r=jya
https://hg.mozilla.org/integration/autoland/rev/63dd1f7a3ff4
part12 : enable wpts. r=jya
https://hg.mozilla.org/integration/autoland/rev/bef097125232
part13.1 - remove unnecessary elements. r=jya
https://hg.mozilla.org/integration/autoland/rev/0c1ef149fc1d
part13.2 - no need to set pref "media.webvtt.regions.enabled". r=jya
https://hg.mozilla.org/integration/autoland/rev/8d7f304e91f7
part13.3 - create elements in HTML, not in JS. r=jya
https://hg.mozilla.org/integration/autoland/rev/9b78a97b3387
part13.4 : don't use anonymous function. r=jya
https://hg.mozilla.org/integration/autoland/rev/6a593a6deb7e
part13.5 - wait text track element's 'load' event. r=jya
https://hg.mozilla.org/integration/autoland/rev/590e376de09a
part13.6 - refactor tests. r=jya
https://hg.mozilla.org/integration/autoland/rev/4159b0ad4c58
part14 : add a test for 'honor user preferences for automatic text track selection'. r=bzbarsky
https://hg.mozilla.org/integration/autoland/rev/39c66f35a9a7
part15 : return error when we failed to create channel. r=jya
https://hg.mozilla.org/integration/autoland/rev/5a39102e79ad
part16 : turn off the pref 'media.block-autoplay-until-in-foreground' in wpt. r=jgraham
https://hg.mozilla.org/integration/autoland/rev/e210709d41e9
part17 : enable loading for track elements in 'browser_cache.js ' and 'trackerFrame.html '. r=timhuang
Upstream web-platform-tests status checks passed, PR will merge once commit reaches central.

According to the spec step2 [1], now we won't start loading for the track element with disabled text track.

[1] https://html.spec.whatwg.org/multipage/media.html#start-the-track-processing-model

Keywords: dev-doc-needed
Regressions: 1562021

Is this a change that actually needs to be documented? Is the web developer or end user really going to notice this, typically? Feels a lot like an implementation detail, so to speak.

Flags: needinfo?(alwu)

It will be good if we can document it. To let users know that if they want to load a track and do any related stuffs on that track, such as waiting load event, that track should be hidden or showing.

Flags: needinfo?(alwu)

Documentation updates:

There is a little inaccuracy for the TextTrack.mode [1], the mode of the text track, which has default keyword, would only be automatically changed to showing or hidden by executing honor user preferences for automatic text track selection [2]. That selection would only be done ONCE, so if you add default on a diabled track after finished the selection, the track mode would not be changed.

[1] https://wiki.developer.mozilla.org/en-US/docs/Web/API/TextTrack/mode
[2] https://html.spec.whatwg.org/multipage/media.html#honor-user-preferences-for-automatic-text-track-selection

Flags: needinfo?(eshepherd)
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: