Myk Melez [:myk] [@mykmelez] Reporter
	Description • 7 years ago

Marionett's test_switch_window_chrome.py test script fails consistently when I run it locally in headless mode on Mac.  The specific test that fails is TestSwitchWindowChrome.test_switch_tabs_for_new_background_window_without_focus_change:

$ MOZ_HEADLESS=1 ./mach marionette-test ./testing/marionette/harness/marionette_harness/tests/unit/test_switch_window_chrome.py
warning: ./mach marionette-test is deprecated; please use ./mach marionette test
 0:00.00 LOG: MainThread INFO Using workspace for temporary data: "/Users/myk/Projects/gecko"
 0:00.00 LOG: MainThread mozversion INFO application_buildid: 20170801094618
 0:00.01 LOG: MainThread mozversion INFO application_display_name: Nightly
 0:00.01 LOG: MainThread mozversion INFO application_id: {ec8030f7-c20a-464f-9b0e-13a3a9e97384}
 0:00.01 LOG: MainThread mozversion INFO application_name: Firefox
 0:00.01 LOG: MainThread mozversion INFO application_remotingname: firefox
 0:00.01 LOG: MainThread mozversion INFO application_vendor: Mozilla
 0:00.01 LOG: MainThread mozversion INFO application_version: 56.0a1
 0:00.01 LOG: MainThread mozversion INFO platform_buildid: 20170801094618
 0:00.01 LOG: MainThread mozversion INFO platform_version: 56.0a1
 0:00.01 LOG: MainThread INFO Application command: /Users/myk/Projects/gecko/obj-x86_64-apple-darwin15.6.0/dist/Nightly.app/Contents/MacOS/firefox -no-remote -marionette -foreground -profile /var/folders/lp/8t_7y24119720_hjp5wc_4cr0000gn/T/tmpPK_EJI.mozrunner
 0:02.82 LOG: MainThread INFO Profile path is /var/folders/lp/8t_7y24119720_hjp5wc_4cr0000gn/T/tmpPK_EJI.mozrunner
 0:02.82 LOG: MainThread INFO Starting fixture servers
 0:03.20 LOG: MainThread INFO Fixture server listening on http://127.0.0.1:60211/
 0:03.20 LOG: MainThread INFO Fixture server listening on https://127.0.0.1:60212/
 0:03.44 LOG: MainThread INFO e10s is enabled
 0:03.46 LOG: MainThread mozversion INFO application_buildid: 20170801094618
 0:03.46 LOG: MainThread mozversion INFO application_display_name: Nightly
 0:03.46 LOG: MainThread mozversion INFO application_id: {ec8030f7-c20a-464f-9b0e-13a3a9e97384}
 0:03.46 LOG: MainThread mozversion INFO application_name: Firefox
 0:03.46 LOG: MainThread mozversion INFO application_remotingname: firefox
 0:03.46 LOG: MainThread mozversion INFO application_vendor: Mozilla
 0:03.46 LOG: MainThread mozversion INFO application_version: 56.0a1
 0:03.46 LOG: MainThread mozversion INFO platform_buildid: 20170801094618
 0:03.46 LOG: MainThread mozversion INFO platform_version: 56.0a1
 0:03.46 SUITE_START: MainThread 1
 0:03.47 TEST_START: MainThread test_switch_window_content.py TestSwitchToWindowContent.test_switch_from_content_to_chrome_window_should_not_change_selected_tab
 0:04.23 TEST_END: MainThread PASS
 0:04.24 TEST_START: MainThread test_switch_window_content.py TestSwitchToWindowContent.test_switch_tabs_with_focus_change
 0:04.80 TEST_END: MainThread PASS
 0:04.80 TEST_START: MainThread test_switch_window_content.py TestSwitchToWindowContent.test_switch_tabs_without_focus_change
 0:05.33 TEST_END: MainThread PASS
 0:05.33 TEST_START: MainThread test_switch_window_content.py TestSwitchToWindowContent.test_switch_to_new_private_browsing_window_has_to_register_browsers
 0:06.16 TEST_END: MainThread PASS
 0:06.16 TEST_START: MainThread test_switch_window_chrome.py TestSwitchWindowChrome.test_switch_from_content_to_chrome_window_should_not_change_selected_tab
 0:06.53 TEST_END: MainThread PASS
 0:06.53 TEST_START: MainThread test_switch_window_chrome.py TestSwitchWindowChrome.test_switch_tabs_for_new_background_window_without_focus_change
 0:07.45 TEST_END: MainThread FAIL, expected PASS
Traceback (most recent call last):
  File "/Users/myk/Projects/gecko/testing/marionette/harness/marionette_harness/marionette_test/testcases.py", line 156, in run
    testMethod()
  File "/Users/myk/Projects/gecko/testing/marionette/harness/marionette_harness/tests/unit/test_switch_window_chrome.py", line 58, in test_switch_tabs_for_new_background_window_without_focus_change
    self.assertEqual(self.get_selected_tab_index(), second_tab_index)
AssertionError: 0 != 1
 0:07.59 TEST_START: MainThread test_switch_window_chrome.py TestSwitchWindowChrome.test_switch_tabs_for_new_foreground_window_with_focus_change
 0:08.60 TEST_END: MainThread PASS
 0:08.60 TEST_START: MainThread test_switch_window_chrome.py TestSwitchWindowChrome.test_switch_tabs_for_new_foreground_window_without_focus_change
 0:09.77 TEST_END: MainThread PASS
 0:09.77 TEST_START: MainThread test_switch_window_chrome.py TestSwitchWindowChrome.test_switch_tabs_with_focus_change
 0:10.28 TEST_END: MainThread PASS
 0:10.28 TEST_START: MainThread test_switch_window_chrome.py TestSwitchWindowChrome.test_switch_tabs_without_focus_change
 0:10.79 TEST_END: MainThread PASS
 0:10.79 TEST_START: MainThread test_switch_window_chrome.py TestSwitchWindowChrome.test_switch_to_new_private_browsing_window_has_to_register_browsers
 0:11.71 TEST_END: MainThread PASS
 0:11.71 LOG: MainThread INFO 
SUMMARY
-------
 0:11.71 LOG: MainThread INFO passed: 10
 0:11.71 LOG: MainThread INFO failed: 1
 0:11.71 LOG: MainThread INFO todo: 0
 0:11.71 LOG: MainThread INFO 
FAILED TESTS
-------
 0:11.71 LOG: MainThread INFO test_switch_window_chrome.py test_switch_window_chrome.TestSwitchWindowChrome.test_switch_tabs_for_new_background_window_without_focus_change
 0:11.71 SUITE_END: MainThread 
Summary
=======

Ran 11 tests
Expected results: 10
Unexpected results: 1 (FAIL: 1)

Unexpected Results
==================

FAIL test_switch_window_chrome.py TestSwitchWindowChrome.test_switch_tabs_for_new_background_window_without_focus_change

	Brendan Dahl [:bdahl]
	Comment 1 • 7 years ago

This test is also currently disabled (see dup bug). This will be fixed by bug 1373739.

	Brendan Dahl [:bdahl]
	Comment 2 • 7 years ago

I just realized why you're seeing this...For marionette please use the --headless command line switch so those tests are skipped.

	Myk Melez [:myk] [@mykmelez] Reporter
	Comment 3 • 7 years ago

(In reply to Brendan Dahl [:bdahl] from comment #2)
> I just realized why you're seeing this...For marionette please use the
> --headless command line switch so those tests are skipped.

Indeed!  For posterity, I was running tests like this:

MOZ_HEADLESS=1 ./mach marionette-test

Whereas I should have been running tests like this:

./mach marionette-test --headless

	Andreas Tolfsen ❲:ato❳
	Comment 4 • 7 years ago

(In reply to Myk Melez [:myk] [@mykmelez] from comment #3)

> (In reply to Brendan Dahl [:bdahl] from comment #2)
> 
> > I just realized why you're seeing this...For marionette please
> > use the --headless command line switch so those tests are
> > skipped.
> 
> Indeed!  For posterity, I was running tests like this:
> 
> 	MOZ_HEADLESS=1 ./mach marionette-test
> 
> Whereas I should have been running tests like this:
> 
> 	./mach marionette-test --headless

It would be nice if the mach command would imply --headless
if MOZ_HEADLESS was set in the environment.  I filed
https://bugzilla.mozilla.org/show_bug.cgi?id=1386581 about this.