Closed Bug 974471 Opened 10 years ago Closed 6 years ago

Expected failures in setUp/tearDown are counted as errors

Categories

(Remote Protocol :: Marionette, defect, P3)

x86
macOS
defect

Tracking

(Not tracked)

RESOLVED WONTFIX

People

(Reporter: ato, Unassigned)

Details

(Keywords: pi-marionette-runner, Whiteboard: [runner])

From bsilverberg's comment in bug 941158:

I'm not sure that this has been resolved properly. Currently, if a test is marked as an expected failure and it errors during tearDown it gets reported as both an expected failure and an error. I believe we should not be getting the latter.

Here is a sample stack trace from a job [1] on our CI:

test_import_contacts_from_gmail (test_import_contacts_from_gmail.TestImportContactsFromGmail) ... expected failure
ERROR

======================================================================
ERROR: None
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.ui.adhoc/.env/local/lib/python2.7/site-packages/marionette_client-0.7.3-py2.7.egg/marionette/marionette_test.py", line 173, in run
    self.tearDown()
  File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.ui.adhoc/tests/python/gaia-ui-tests/gaiatest/gaia_test.py", line 1119, in tearDown
    MarionetteTestCase.tearDown(self)
  File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.ui.adhoc/.env/local/lib/python2.7/site-packages/marionette_client-0.7.3-py2.7.egg/marionette/marionette_test.py", line 318, in tearDown
    self.marionette.set_context("content")
  File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.ui.adhoc/.env/local/lib/python2.7/site-packages/marionette_client-0.7.3-py2.7.egg/marionette/marionette.py", line 847, in set_context
    return self._send_message('setContext', 'ok', value=context)
  File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.ui.adhoc/.env/local/lib/python2.7/site-packages/marionette_client-0.7.3-py2.7.egg/marionette/marionette.py", line 581, in _send_message
    raise MarionetteException(message="Please start a session")
MarionetteException: MarionetteException: Please start a session
TEST-UNEXPECTED-FAIL | test_import_contacts_from_gmail.py test_import_contacts_from_gmail.TestImportContactsFromGmail.test_import_contacts_from_gmail | 
----------------------------------------------------------------------
Ran 1 test in 418.978s

FAILED (errors=1, expected failures=1)


[1] http://qa-selenium.mv.mozilla.com:8080/job/b2g.hamachi.mozilla-central.ui.adhoc/199/
Whiteboard: [runner]
[mass update] Setting Harness bugs to all P3
Priority: -- → P3
Unlikely to ever be fixed.
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → WONTFIX
Product: Testing → Remote Protocol
You need to log in before you can comment on or make changes to this bug.