Andreas Tolfsen ❲:ato❳ Assignee
	Description • 6 years ago

After the recent upgrade of pytest upstream that yesterday landed
in m-c, the WPT wdspec tests outputs a percentage progress number
of how far along it is running a test.

This is misleading because we invoke pytest on a per-file level which
means the percentage is only accurate on for one test file at a time,
but more importantly it causes a lot more output which is distracting:

>  0:12.08 INFO STDOUT: testing/web-platform/tests/webdriver/tests/element_click/select.py::test_click_option 
>  0:12.09 pid:27064 1515590214026	mozrunner::runner	INFO	Running command: "/home/ato/src/gecko/obj-x86_64-pc-linux-gnu/dist/bin/firefox" "-marionette" "-profile" "/tmp/rust_mozprofile.yIWe3T4R3eWL"
>  0:12.12 pid:27064 *** You are running in headless mode.
>  0:12.46 pid:27064 1515590214394	Marionette	INFO	Enabled via --marionette
>  0:13.67 pid:27064 1515590215609	Marionette	INFO	Listening on port 2828
>  0:14.21 INFO STDOUT: PASSED
>  0:14.21 INFO STDOUT:  [  9%]
>  0:14.29 INFO STDOUT: testing/web-platform/tests/webdriver/tests/element_click/select.py::test_click_multiple_option 
>  0:14.62 INFO STDOUT: PASSED
>  0:14.62 INFO STDOUT:  [ 18%]
>  0:14.67 INFO STDOUT: testing/web-platform/tests/webdriver/tests/element_click/select.py::test_click_preselected_option 
>  0:15.23 INFO STDOUT: PASSED
>  0:15.23 INFO STDOUT:  [ 27%]
>  0:15.27 INFO STDOUT: testing/web-platform/tests/webdriver/tests/element_click/select.py::test_click_preselected_multiple_option 
>  0:15.85 INFO STDOUT: PASSED
>  0:15.85 INFO STDOUT:  [ 36%]
>  0:15.89 INFO STDOUT: testing/web-platform/tests/webdriver/tests/element_click/select.py::test_click_deselects_others 
>  0:16.86 INFO STDOUT: PASSED
>  0:16.86 INFO STDOUT:  [ 45%]
>  0:16.93 INFO STDOUT: testing/web-platform/tests/webdriver/tests/element_click/select.py::test_click_multiple_does_not_deselect_others 
>  0:17.68 INFO STDOUT: PASSED
>  0:17.68 INFO STDOUT:  [ 54%]
>  0:17.72 INFO STDOUT: testing/web-platform/tests/webdriver/tests/element_click/select.py::test_click_selected_option 
>  0:18.25 INFO STDOUT: PASSED
>  0:18.25 INFO STDOUT:  [ 63%]
>  0:18.29 INFO STDOUT: testing/web-platform/tests/webdriver/tests/element_click/select.py::test_click_selected_multiple_option 
>  0:18.81 INFO STDOUT: PASSED
>  0:18.81 INFO STDOUT:  [ 72%]
>  0:18.84 INFO STDOUT: testing/web-platform/tests/webdriver/tests/element_click/select.py::test_out_of_view_dropdown 
>  0:19.14 INFO STDOUT: PASSED
>  0:19.14 INFO STDOUT:  [ 81%]
>  0:19.17 INFO STDOUT: testing/web-platform/tests/webdriver/tests/element_click/select.py::test_out_of_view_multiple 
>  0:19.46 INFO STDOUT: PASSED
>  0:19.46 INFO STDOUT:  [ 90%]
>  0:19.50 INFO STDOUT: testing/web-platform/tests/webdriver/tests/element_click/select.py::test_option_disabled 
>  0:19.81 INFO STDOUT: PASSED
>  0:19.81 INFO STDOUT:  [100%]
>  0:19.85 INFO STDOUT: ========================== 11 passed in 7.78 seconds ===========================
>  0:19.88 TEST_END: Test OK. Subtests passed 11/11. Unexpected 0

	Andreas Tolfsen ❲:ato❳ Assignee
	Comment 1 • 6 years ago

Do we know what the pytest option is to disable this new “feature”?

	Dave Hunt [:davehunt] [he/him] ⌚BST
	Comment 2 • 6 years ago

You can set console_output_style = classic in the config file, or use -o console_output_style=classic on the command line.

	Andreas Tolfsen ❲:ato❳ Assignee
	Comment 3 • 6 years ago

(In reply to Dave Hunt (:davehunt) from comment #2)

> You can set console_output_style = classic in the config file, or
> use -o console_output_style=classic on the command line.

Unfortunately I cannot get this working with the way we invoke
pytest.main():

> pytest.main(["-o", "console_output_style=classic"])

Even with variations of that I still keep getting this error:

> 0:03.55 INFO STDERR: ERROR: -o/--override-ini expects option=value style.

	Dave Hunt [:davehunt] [he/him] ⌚BST
	Comment 4 • 6 years ago

I've raised https://github.com/pytest-dev/pytest/issues/3103 to address this in pytest, however I believe passing as follows should work for us in the meantime:

> pytest.main(["-o=console_output_style=classic"])

	Andreas Tolfsen ❲:ato❳ Assignee
	Comment 5 • 6 years ago

(In reply to Dave Hunt (:davehunt) from comment #4)
> […] however I believe passing as follows should work for us in the
> meantime:
> 
> > pytest.main(["-o=console_output_style=classic"])

I can confirm that this works.

	Dave Hunt [:davehunt] [he/him] ⌚BST
	Comment 7 • 6 years ago
mozreview-review

Comment on attachment 8941463 [details]
Bug 1429388 - Use classic pytest output style for WPT.

https://reviewboard.mozilla.org/r/211746/#review217534

	James Graham [:jgraham]
	Comment 8 • 6 years ago
mozreview-review

Comment on attachment 8941463 [details]
Bug 1429388 - Use classic pytest output style for WPT.

https://reviewboard.mozilla.org/r/211746/#review217576

	Pulsebot
	Comment 9 • 6 years ago

Pushed by atolfsen@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/4f065bc82a46
Use classic pytest output style for WPT. r=davehunt,jgraham

	Dorel Luca [:dluca]
	Comment 10 • 6 years ago
bugherder

https://hg.mozilla.org/mozilla-central/rev/4f065bc82a46