Jonathan Griffin (:jgriffin) Reporter
	Description • 11 years ago

It appears that when a test is marked as an expected failure in the manifest, this status isn't honored when the test fails in either setUp or tearDown, causing the test to be marked as an unexpected failure.

A couple of cases in point from a gaia-ui-tests run:

06:53:47 TEST-START test_import_contacts_from_gmail.py
06:58:44 test_import_contacts_from_gmail (test_import_contacts_from_gmail.TestImportContactsFromGmail) ... expected failure
06:58:44 ERROR
06:58:44 
06:58:44 ======================================================================
06:58:44 ERROR: None
06:58:44 ----------------------------------------------------------------------
06:58:44 Traceback (most recent call last):
06:58:44   File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.ui/.env/local/lib/python2.7/site-packages/marionette_client-0.6.2-py2.7.egg/marionette/marionette_test.py", line 173, in run
06:58:44     self.tearDown()
06:58:44   File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.ui/tests/python/gaia-ui-tests/gaiatest/gaia_test.py", line 759, in tearDown
06:58:44     MarionetteTestCase.tearDown(self)
06:58:44   File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.ui/.env/local/lib/python2.7/site-packages/marionette_client-0.6.2-py2.7.egg/marionette/marionette_test.py", line 318, in tearDown
06:58:44     self.marionette.set_context("content")
06:58:44   File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.ui/.env/local/lib/python2.7/site-packages/marionette_client-0.6.2-py2.7.egg/marionette/marionette.py", line 803, in set_context
06:58:44     return self._send_message('setContext', 'ok', value=context)
06:58:44   File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.ui/.env/local/lib/python2.7/site-packages/marionette_client-0.6.2-py2.7.egg/marionette/marionette.py", line 553, in _send_message
06:58:44     raise MarionetteException(message="Please start a session")
06:58:44 TEST-UNEXPECTED-FAIL | test_import_contacts_from_gmail.py test_import_contacts_from_gmail.TestImportContactsFromGmail.test_import_contacts_from_gmail | MarionetteException: Please start a session

08:23:41 TEST-START test_settings_airplane_mode.py
08:25:29 test_toggle_airplane_mode (test_settings_airplane_mode.TestAirplaneMode) ... ERROR
08:25:29 
08:25:29 ======================================================================
08:25:29 ERROR: None
08:25:29 ----------------------------------------------------------------------
08:25:29 Traceback (most recent call last):
08:25:29   File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.ui/.env/local/lib/python2.7/site-packages/marionette_client-0.6.2-py2.7.egg/marionette/marionette_test.py", line 127, in run
08:25:29     self.setUp()
08:25:29   File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.ui/tests/python/gaia-ui-tests/gaiatest/tests/functional/settings/test_settings_airplane_mode.py", line 15, in setUp
08:25:29     self.data_layer.connect_to_wifi()
08:25:29   File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.ui/tests/python/gaia-ui-tests/gaiatest/gaia_test.py", line 298, in connect_to_wifi
08:25:29     self.enable_wifi()
08:25:29   File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.ui/tests/python/gaia-ui-tests/gaiatest/gaia_test.py", line 287, in enable_wifi
08:25:29     result = self.marionette.execute_async_script("return GaiaDataLayer.enableWiFi()", special_powers=True)
08:25:29   File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.ui/.env/local/lib/python2.7/site-packages/marionette_client-0.6.2-py2.7.egg/marionette/marionette.py", line 1073, in execute_async_script
08:25:29     filename=os.path.basename(frame[0]))
08:25:29   File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.ui/.env/local/lib/python2.7/site-packages/marionette_client-0.6.2-py2.7.egg/marionette/marionette.py", line 577, in _send_message
08:25:29     self._handle_error(response)
08:25:29   File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.ui/.env/local/lib/python2.7/site-packages/marionette_client-0.6.2-py2.7.egg/marionette/marionette.py", line 626, in _handle_error
08:25:29     raise ScriptTimeoutException(message=message, status=status, stacktrace=stacktrace)
08:25:29 TEST-UNEXPECTED-FAIL | test_settings_airplane_mode.py test_settings_airplane_mode.TestAirplaneMode.test_toggle_airplane_mode | ScriptTimeoutException: timed out

	Zac C (:zac)
	Comment 1 • 11 years ago

mdas asked me to mark up important Marionette issues for Q1

	Andreas Tolfsen ❲:ato❳ Assignee
	Comment 2 • 10 years ago

Attaching a patch that solves the problem, but solves it in a suboptimal way due to the existing design of the function.  I've filed bug 967566 about refactoring this to reduce code duplication and complexity.

try: https://tbpl.mozilla.org/?tree=Try&rev=af64e5347ad4

	Zac C (:zac)
	Comment 3 • 10 years ago

Comment on attachment 8370123 [details] [diff] [review]
0001-Bug-941158-Honour-expected-failures-in-setUp-or-tear.patch

Actually now I see it I don't think I'm the right one to r? this. I will push it back to jgriffin who originally authored the code.

	Jonathan Griffin (:jgriffin) Reporter
	Comment 4 • 10 years ago

Comment on attachment 8370123 [details] [diff] [review]
0001-Bug-941158-Honour-expected-failures-in-setUp-or-tear.patch

Review of attachment 8370123 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks, lgtm!

	Jonathan Griffin (:jgriffin) Reporter
	Comment 5 • 10 years ago

And thanks for filing bug 967566.

	Ryan VanderMeulen [:RyanVM]
	Comment 6 • 10 years ago

https://hg.mozilla.org/integration/mozilla-inbound/rev/a246bbc0c61c

	Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)
	Comment 7 • 10 years ago

Backed out in http://hg.mozilla.org/integration/mozilla-inbound/rev/54c21f290a8d for breaking Marionette: https://tbpl.mozilla.org/php/getParsedLog.php?id=34165579&tree=Mozilla-Inbound

	Andreas Tolfsen ❲:ato❳ Assignee
	Comment 8 • 10 years ago

Attaching revised patch which fixes problem.  Carrying on r+ from jgriffin.

	Andreas Tolfsen ❲:ato❳ Assignee
	Comment 9 • 10 years ago

Fixed wrong reviewer in patch commit message.

To be on the safe side I've triggered another try run: https://tbpl.mozilla.org/?tree=Try&rev=8edcde22e53e

	Andreas Tolfsen ❲:ato❳ Assignee
	Comment 10 • 10 years ago

Got a green Gu in latest try run.  Due to bug 967826 Gu on Linux has a ~90% failure rate at the moment, so I'm taking this as a sign that things are OK with the revised patch.

	Ryan VanderMeulen [:RyanVM]
	Comment 11 • 10 years ago

https://hg.mozilla.org/integration/mozilla-inbound/rev/799c9a7716ef

	Carsten Book [:Tomcat]
	Comment 12 • 10 years ago

https://hg.mozilla.org/mozilla-central/rev/799c9a7716ef

	Bob Silverberg [:bsilverberg]
	Comment 13 • 10 years ago

I'm not sure that this has been resolved properly. Currently, if a test is marked as an expected failure and it errors during tearDown it gets reported as both an expected failure and an error. I believe we should not be getting the latter.

Here is a sample stack trace from a job [1] on our CI:

test_import_contacts_from_gmail (test_import_contacts_from_gmail.TestImportContactsFromGmail) ... expected failure
ERROR

======================================================================
ERROR: None
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.ui.adhoc/.env/local/lib/python2.7/site-packages/marionette_client-0.7.3-py2.7.egg/marionette/marionette_test.py", line 173, in run
    self.tearDown()
  File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.ui.adhoc/tests/python/gaia-ui-tests/gaiatest/gaia_test.py", line 1119, in tearDown
    MarionetteTestCase.tearDown(self)
  File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.ui.adhoc/.env/local/lib/python2.7/site-packages/marionette_client-0.7.3-py2.7.egg/marionette/marionette_test.py", line 318, in tearDown
    self.marionette.set_context("content")
  File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.ui.adhoc/.env/local/lib/python2.7/site-packages/marionette_client-0.7.3-py2.7.egg/marionette/marionette.py", line 847, in set_context
    return self._send_message('setContext', 'ok', value=context)
  File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.ui.adhoc/.env/local/lib/python2.7/site-packages/marionette_client-0.7.3-py2.7.egg/marionette/marionette.py", line 581, in _send_message
    raise MarionetteException(message="Please start a session")
MarionetteException: MarionetteException: Please start a session
TEST-UNEXPECTED-FAIL | test_import_contacts_from_gmail.py test_import_contacts_from_gmail.TestImportContactsFromGmail.test_import_contacts_from_gmail | 
----------------------------------------------------------------------
Ran 1 test in 418.978s

FAILED (errors=1, expected failures=1)


[1] http://qa-selenium.mv.mozilla.com:8080/job/b2g.hamachi.mozilla-central.ui.adhoc/199/

	Andreas Tolfsen ❲:ato❳ Assignee
	Comment 14 • 10 years ago

Since this patch wasn't backed out, I filed bug 974471 about the problem bsilverberg describes.  Resolving this.