Closed Bug 1465527 Opened 6 years ago Closed 6 years ago

reftest error messages are not propagated to wptrunner

Categories

(Remote Protocol :: Marionette, defect)

defect
Not set
normal

Tracking

(firefox62 fixed)

RESOLVED FIXED
mozilla62
Tracking Status
firefox62 --- fixed

People

(Reporter: ato, Assigned: ato)

References

Details

Attachments

(3 files)

When an internal error occurs in the reftest harness embedded in
Marionette, only the error’s stacktrace is propagated to wptrunner.

The error message should be preserved and passed in the "message"
field (where the stacktrace is passed now).
Assignee: nobody → ato
Status: NEW → ASSIGNED
The attached will produce this output:

>  0:15.93 TEST_START: /2dcontext/building-paths/canvas_complexshapes_arcto_001.htm
>  0:16.60 pid:20481 1527700520246	Marionette	INFO	Testing http://web-platform.test:8000/2dcontext/building-paths/canvas_complexshapes_arcto_001.htm == http://web-platform.test:8000/2dcontext/building-paths/canvas_complexshapes_arcto_001-ref.htm
>  0:16.82 pid:20481 1527700520471	Marionette	INFO	Found 200 pixels different, maximum difference per channel 8
>  0:16.83 TEST_END: ERROR, expected FAIL - RangeError: yoyoyo
> run/testRunner<@chrome://marionette/content/reftest.js:197:15
> async*run@chrome://marionette/content/reftest.js:193:23
> async*GeckoDriver.prototype.runReftest@chrome://marionette/content/driver.js:3493:24
> Async*despatch@chrome://marionette/content/server.js:296:20
> async*execute@chrome://marionette/content/server.js:270:11
> async*onPacket/<@chrome://marionette/content/server.js:245:15
> async*onPacket@chrome://marionette/content/server.js:244:8
> _onJSONObjectReady/<@chrome://marionette/content/transport.js:500:9
Comment on attachment 8981973 [details]
Bug 1465527 - Print stacks from parent tests in mach formatter.

https://reviewboard.mozilla.org/r/247998/#review254098
Attachment #8981973 - Flags: review?(james) → review+
Comment on attachment 8981974 [details]
Bug 1465527 - Log parent test result stacks.

https://reviewboard.mozilla.org/r/248000/#review254100

::: testing/web-platform/tests/tools/wptrunner/wptrunner/executors/base.py:70
(Diff revision 1)
>      def __call__(self, test, result, extra=None):
>          """Convert a JSON result into a (TestResult, [SubtestResult]) tuple"""
>          result_url, status, message, stack, subtest_results = result
>          assert result_url == test.url, ("Got results from %s, expected %s" %
>                                          (result_url, test.url))
>          harness_result = test.result_cls(self.harness_codes[status], message, extra=extra)

Can you add `stack=stack` for testharness tests while you're here?
Attachment #8981974 - Flags: review?(james) → review+
Comment on attachment 8981975 [details]
Bug 1465527 - Return structured stacktrace from Marionette reftest harness.

https://reviewboard.mozilla.org/r/248002/#review254102

Thanks!
Attachment #8981975 - Flags: review?(james) → review+
Pushed by atolfsen@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/72eb0c354c31
Print stacks from parent tests in mach formatter. r=jgraham
https://hg.mozilla.org/integration/autoland/rev/2492931ce122
Log parent test result stacks. r=jgraham
https://hg.mozilla.org/integration/autoland/rev/8fd817cc95a7
Return structured stacktrace from Marionette reftest harness. r=jgraham
Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/11804 for changes under testing/web-platform/tests
Product: Testing → Remote Protocol
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: