Closed Bug 1294075 Opened 8 years ago Closed 5 years ago

Intermittent test_accessibility.py TestAccessibility.test_element_is_not_enabled_to_accessbility | AssertionError: ElementNotAccessibleException not raised

Categories

(Testing :: Marionette Client and Harness, defect, P3)

Version 3
defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: intermittent-bug-filer, Assigned: bdahl)

References

Details

(Keywords: intermittent-failure, Whiteboard: [stockwell disabled])

Attachments

(1 file)

I’m occasionally hitting this when running testing/marionette/harness/marionette_harness/tests/unit/test_accessibility.py on Linux opt locally.  I’m working on bug 1321516 which has to make a few changes to the expectations of the test due to a new ‘element click intercepted’ error introduced by the WebDriver conforming element click implementation.
So it first appeared actually on April 4th (https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=1606323068fa116336bd6109f184b165d778f2c9&selectedJob=88546569). It might be helpful to have a pushlog of all the changes between the nightly builds when this failure appeared.
wow, this really has increased in frequency as of May 22nd:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1294075&startday=2017-05-01&endday=2017-05-30&tree=all

this is almost always occurring on on linux64 opt/pgo and on marionette-headless.


here is a recent log from today:
https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-inbound&job_id=103174842&lineNumber=1354

the failures I see in the log do not appear to be useful:
[task 2017-05-30T21:32:05.577507Z] 21:32:05     INFO - TEST-START | test_accessibility.py TestAccessibility.test_element_is_not_enabled_to_accessbility
[task 2017-05-30T21:32:05.875538Z] 21:32:05     INFO - TEST-UNEXPECTED-FAIL | test_accessibility.py TestAccessibility.test_element_is_not_enabled_to_accessbility | AssertionError: ElementNotAccessibleException not raised
[task 2017-05-30T21:32:05.876444Z] 21:32:05     INFO - Traceback (most recent call last):
[task 2017-05-30T21:32:05.877607Z] 21:32:05     INFO -   File "/home/worker/workspace/build/venv/local/lib/python2.7/site-packages/marionette_harness/marionette_test/testcases.py", line 166, in run
[task 2017-05-30T21:32:05.878211Z] 21:32:05     INFO -     testMethod()
[task 2017-05-30T21:32:05.878838Z] 21:32:05     INFO -   File "/home/worker/workspace/build/tests/marionette/tests/testing/marionette/harness/marionette_harness/tests/unit/test_accessibility.py", line 173, in test_element_is_not_enabled_to_accessbility
[task 2017-05-30T21:32:05.879447Z] 21:32:05     INFO -     lambda element: self.assertRaises(ElementNotAccessibleException,
[task 2017-05-30T21:32:05.880105Z] 21:32:05     INFO -   File "/home/worker/workspace/build/tests/marionette/tests/testing/marionette/harness/marionette_harness/tests/unit/test_accessibility.py", line 94, in run_element_test
[task 2017-05-30T21:32:05.880181Z] 21:32:05     INFO -     testFn(element)
[task 2017-05-30T21:32:05.880731Z] 21:32:05     INFO -   File "/home/worker/workspace/build/tests/marionette/tests/testing/marionette/harness/marionette_harness/tests/unit/test_accessibility.py", line 174, in <lambda>
[task 2017-05-30T21:32:05.881271Z] 21:32:05     INFO -     element.is_enabled))
[task 2017-05-30T21:32:05.881851Z] 21:32:05     INFO - TEST-INFO took 217ms

:whimboo can we fix this issue this week or consider disabling this test?  The failure rate is too high to provide value while running this test.
Flags: needinfo?(hskupin)
Whiteboard: [stockwell needswork]
It's not up to me. Brendan is working on some improvements for Marionette headless tests, so lets ask him.
Flags: needinfo?(hskupin) → needinfo?(bdahl)
This appears to be the same issue as bug 1341493. I'll disable on headless for now, but the fix in https://bugzilla.mozilla.org/show_bug.cgi?id=1341493#c28 is the real solution. Also, as I mentioned in that bug, the issue appears to not really be headless but some of the accessibility code, so this won't be a high priority for me.
Flags: needinfo?(bdahl)
Comment on attachment 8874513 [details] [diff] [review]
Disable marionette accessibility test on headless.

Review of attachment 8874513 [details] [diff] [review]:
-----------------------------------------------------------------

A mozreview patch would have been nice so I could have pushed it immediately. So please do so yourself, once you are online. Thanks.
Attachment #8874513 - Flags: review?(hskupin) → review+
Pushed by bdahl@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/1b21c086b953
Disable marionette accessibility test on headless.
Whiteboard: [stockwell needswork] → [stockwell disabled]
Hopefully bug 1341493 fixed this one as well. Early OrangeFactor results are looking promising.
Priority: -- → P3
The leave-open keyword is there and there is no activity for 6 months.
:automatedtester, maybe it's time to close this bug?
Flags: needinfo?(dburns)
This appears to have been reenabled
Status: NEW → RESOLVED
Closed: 5 years ago
Flags: needinfo?(dburns)
Resolution: --- → FIXED
Assignee: nobody → bdahl
This was fixed as part of bug 1341493.
Depends on: 1341493
Product: Testing → Remote Protocol
Moving bug to Testing::Marionette Client and Harness component per bug 1815831.
Component: Marionette → Marionette Client and Harness
Product: Remote Protocol → Testing
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: