Closed
Bug 1354750
Opened 7 years ago
Closed 7 years ago
Intermittent TEST-UNEXPECTED-TIMEOUT | /webdriver/actions/key.py | expected OK
Categories
(Testing :: web-platform-tests, defect)
Tracking
(firefox55 fixed, firefox56 fixed)
RESOLVED
FIXED
mozilla56
People
(Reporter: intermittent-bug-filer, Assigned: jgraham)
References
Details
(Keywords: intermittent-failure, Whiteboard: [stockwell fixed:other])
Attachments
(1 file)
Filed by: philringnalda [at] gmail.com https://treeherder.mozilla.org/logviewer.html#?job_id=89764655&repo=autoland https://queue.taskcluster.net/v1/task/GsDioRxQQmyda5fcPKX2Tw/runs/0/artifacts/public/logs/live_backing.log
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment 7•7 years ago
|
||
:jgraham- can you look at this given that this is failing often every day.
Flags: needinfo?(james)
Whiteboard: [stockwell needswork]
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Assignee | ||
Comment 11•7 years ago
|
||
It looks like maybe when this fails the test file isn't loading at all? Otherwise I would expect some intermediate output. Also increasing the timeout doesn't help. If that's true disabling it *might* just make the following test intermittent in the same way.
Comment hidden (Intermittent Failures Robot) |
Assignee | ||
Comment 13•7 years ago
|
||
Something bad happens in pytest startup so we never start running tests. I'm pretty sure disabling won't have any impact here except to change the test that's unstable. I'm doing an excessie number of try runs to figure out what's up.
Flags: needinfo?(james)
Assignee | ||
Updated•7 years ago
|
Flags: needinfo?(james)
Assignee | ||
Comment 14•7 years ago
|
||
So so there's good news and bad news. The good news is I think I know what the problem is. The bad news is what the problem is. So I think that what's happening here is a nasty interaction between multiprocessing and threading. On Linux multiprocessing calls fork() which provides a child process with a copy (well copy-on-write) of the parent process memory. That usually works fine. However in the case that you have a mutex in the parent process that happens to be locked at the time of the fork, and you attempt to acquire that same mutex in the child process it will now always appear locked because even if it were released in the parent that change is never reflected in the child's copy. In our case we see an intermittent deadlock when trying to set up the structured logger from pytest, despite the fact that there's only a single thread in that process. Keying the lock itself on pid makes the problem go away, but isn't a very general solution because there could be other bits of code that will fail in the same way at some later time. I'm not sure what the best solution here is other than "don't use threading and multiprocessing together" (needless to say I didn't know about this issue when the code was written). Possibly landing some hack in the short term is a good idea.
Comment hidden (Intermittent Failures Robot) |
Comment 16•7 years ago
|
||
I am glad there is some news about this- do you think we can get a hack in this week, or should we disable this test (or wdspec) job until we can get a hack landed?
Assignee | ||
Comment 17•7 years ago
|
||
Disabling this test won't help at all. Disabling the entire suite would be bad for other reasons. You would have to make the job Tier < 1. But I have an idea for a better solution that should be low effort (basically there's a backported version of the Python 3 multiprocessing module that allows you to use multiple processes in a way that doesn't hillariously violate posix semantics), so I'll try that.
Comment hidden (mozreview-request) |
Comment hidden (Intermittent Failures Robot) |
Comment 20•7 years ago
|
||
mozreview-review |
Comment on attachment 8880151 [details] Bug 1354750 - Disable loading mozlog plugin with pytest for wpt, https://reviewboard.mozilla.org/r/151526/#review156614
Attachment #8880151 -
Flags: review?(ato) → review+
Comment hidden (mozreview-request) |
Comment 22•7 years ago
|
||
Pushed by james@hoppipolla.co.uk: https://hg.mozilla.org/integration/autoland/rev/a7e6d0b5fbdd Disable loading mozlog plugin with pytest for wpt, r=ato
Assignee | ||
Updated•7 years ago
|
Flags: needinfo?(james)
Comment 23•7 years ago
|
||
bugherder |
https://hg.mozilla.org/mozilla-central/rev/a7e6d0b5fbdd
Status: NEW → RESOLVED
Closed: 7 years ago
status-firefox56:
--- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla56
Comment 24•7 years ago
|
||
should we uplift this to beta?
Whiteboard: [stockwell needswork] → [stockwell fixed:other]
Updated•7 years ago
|
Assignee: nobody → james
Comment 25•7 years ago
|
||
bugherder uplift |
https://hg.mozilla.org/releases/mozilla-beta/rev/b4bd21b3d861
status-firefox55:
--- → fixed
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
You need to log in
before you can comment on or make changes to this bug.
Description
•