Closed Bug 1202375 Opened 9 years ago Closed 8 years ago

Intermittent | test_direct_update.py TestDirectUpdate.test_update, test_fallback_update.py TestFallbackUpdate.test_update | AssertionError: Timed out waiting for port!

Categories

(Testing :: Firefox UI Tests, defect)

Product:
Testing
Component:
Firefox UI Tests
Version:
43 Branch
Platform:
All
macOS
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(firefox42 wontfix, firefox43 wontfix, firefox45 ?, firefox46 affected, firefox47 affected, firefox48 affected)

Status:
RESOLVED WONTFIX
Tracking Flags:
Tracking Status
firefox42 --- wontfix
firefox43 --- wontfix
firefox45 --- ?
firefox46 --- affected
firefox47 --- affected
firefox48 --- affected

People

(Reporter: whimboo, Unassigned)

References

Details

(Keywords: intermittent-failure, Whiteboard: [mm-osx-106-3])

Attachments

(1 file)

minidump.dmp
44.16 KB, application/octet-stream
Details 
Due to some reason there is a crash during out update tests. I will have to get it analyzed. Here the stack. Interesting is that Marionette dies with waiting for port but does not see a crash.

08:11:37 JavaScript error: resource://gre/modules/BookmarkHTMLUtils.jsm, line 884: NS_ERROR_UNEXPECTED: Component returned failure code: 0x8000ffff (NS_ERROR_UNEXPECTED) [nsINavBookmarksService.removeFolderChildren]
08:12:39  1:04.17 CRASH: MainThread pid:1580. Test:test_direct_update.py TestDirectUpdate.test_update. Minidump anaylsed:False. Signature:[None]
08:12:39 Crash dump filename: c:\users\mozauto\appdata\local\temp\tmpm9ptkm.mozrunner\minidumps\e63a6a87-fbd0-4d4b-91c0-351bb4785c5b.dmp
08:12:39 No symbols path given, can't process dump.
08:12:39 MINIDUMP_STACKWALK not set, can't process dump.
08:12:39 
08:12:39  1:04.19 LOG: MainThread WARNING Failed to gather test failure debug.
08:12:39 Traceback (most recent call last):
08:12:39 
08:12:39   File "c:\jenkins\workspace\mozilla-central_update\venv\lib\site-packages\marionette\runner\base.py", line 529, in gather_debug
08:12:39     with marionette.using_context(marionette.CONTEXT_CHROME):
08:12:39 
08:12:39   File "C:\Python27\Lib\contextlib.py", line 17, in __enter__
08:12:39     return self.gen.next()
08:12:39 
08:12:39   File "c:\jenkins\workspace\mozilla-central_update\venv\lib\site-packages\marionette_driver\marionette.py", line 1212, in using_context
08:12:39     scope = self._send_message("getContext", key="value")
08:12:39 
08:12:39   File "c:\jenkins\workspace\mozilla-central_update\venv\lib\site-packages\marionette_driver\decorators.py", line 36, in _
08:12:39     return func(*args, **kwargs)
08:12:39 
08:12:39   File "c:\jenkins\workspace\mozilla-central_update\venv\lib\site-packages\marionette_driver\marionette.py", line 681, in _send_message
08:12:39     resp = self.client.send(packet)
08:12:39 
08:12:39   File "c:\jenkins\workspace\mozilla-central_update\venv\lib\site-packages\marionette_transport\transport.py", line 96, in send
08:12:39     self.connect()
08:12:39 
08:12:39   File "c:\jenkins\workspace\mozilla-central_update\venv\lib\site-packages\marionette_transport\transport.py", line 79, in connect
08:12:39     self.sock.connect((self.addr, self.port))
08:12:39 
08:12:39   File "C:\Python27\Lib\socket.py", line 224, in meth
08:12:39     return getattr(self._sock,name)(*args)
08:12:39 
08:12:39 error: [Errno 10061] No connection could be made because the target machine actively refused it
08:12:39 
08:12:39  1:04.19 TEST_END: MainThread ERROR, expected PASS
08:12:39 Traceback (most recent call last):
08:12:39   File "c:\jenkins\workspace\mozilla-central_update\venv\lib\site-packages\marionette\marionette_test.py", line 277, in run
08:12:39     self.setUp()
08:12:39   File "c:\jenkins\workspace\mozilla-central_update\venv\lib\site-packages\firefox_ui_tests\update\direct\test_direct_update.py", line 11, in setUp
08:12:39     UpdateTestCase.setUp(self, is_fallback=False)
08:12:39   File "c:\jenkins\workspace\mozilla-central_update\venv\lib\site-packages\firefox_ui_harness\testcases\update.py", line 62, in setUp
08:12:39     self.restart()
08:12:39   File "c:\jenkins\workspace\mozilla-central_update\venv\lib\site-packages\firefox_ui_harness\testcases\base.py", line 33, in restart
08:12:39     self.marionette.restart(in_app=True)
08:12:39   File "c:\jenkins\workspace\mozilla-central_update\venv\lib\site-packages\marionette_driver\marionette.py", line 986, in restart
08:12:39     assert(self.wait_for_port()), "Timed out waiting for port!"
08:12:39 AssertionError: Timed out waiting for port!
08:12:39
Bug 1201884 shows the same warning but without the crash. Not sure if that is related.
status-firefox42: --- → affected
OS: Windows 8 → Windows
Hardware: Unspecified → All
See Also: → 1201884
There is no way to get crash reports from those machines because they are not stored at the central location under %appdata%/Mozilla/Firefox/Crash Reports, but in the profile which actually gets deleted by Marionette after the tests finished. :(

Crash dump filename: c:\users\mozauto\appdata\local\temp\tmpm9ptkm.mozrunner\minidumps\e63a6a87-fbd0-4d4b-91c0-351bb4785c5b.dmp

I will file a bug for Marionette, so that we can at least copy the minidump files to the %appdata% location in case of symbol handling is not active. It's better then loosing all the information.
Interestingly I was able to hit this exact crash when manually upgrading Firefox on such a box. During the restart of Firefox I got the following crash report:

bp-d5e84fc1-60bb-425b-9e21-5e48a2150907

Which actually routes me to bug 1127270, which is our topcrash bug on Windows right now.
Depends on: 1127270
Depends on: 1202399
(In reply to Henrik Skupin (:whimboo) from comment #6)
> There is no way to get crash reports from those machines because they are
> not stored at the central location under %appdata%/Mozilla/Firefox/Crash
> Reports, but in the profile which actually gets deleted by Marionette after
> the tests finished. :(

This is because we set MOZ_CRASHREPORTER_NO_REPORT in mozrunner to suppress the crash reporter dialog. Sounds like the same thing as bug 1200975.
Well, we don't want the crash reporter window but only keep the minidump files for manually analysis until we can make use of automatic crash analysis via mozcrash (part of my work for the mozharness firefox-ui-tests scripts).
Is there a way to determine whether a content process crash has occurred during these sessions?
Bas, as you can see in the comments above there is no minidump file written at all anymore. Which means I cannot use our update tests to get it reproduced. The one crash I mentioned above (comment 7) I was able to see when manually updating a Firefox version on one of the affected machines. 

If you need more info about the issue regarding the shutdown bug, lets move over to that other bug please.
The underlying crash should be fixed now with the Nightly build from today. I will keep an eye out for crashes during our tests.
Looks like we hit the crash again but keep in mind that we were running builds for the update tests from 2 and 4 days ago! The fix for bug 1127270 went in on Sep 15th which is 3 days ago. I think we should wait for the results starting from Sunday, which will make use of at maximum a build from Sep 16th. If the crash is going away, we can prove that it has been fixed. If not, we might have to wait until we use mozharness on our boxes.
Keywords: intermittent-failure
Whiteboard: [failure]
See Also: → 1222197
Depends on: 1229765
Attached file minidump.dmp — 
Here finally a minidump from that crash as caught by Marionette. Sadly mozcrash is not able to process those minidump files due to bug 474863.
Product: Mozilla QA → Testing
It only happens on the mm-osx-106-3 machine for both the direct and fallback updates.
OS: Windows → Mac OS X
Summary: Intermittent test_direct_update.py TestDirectUpdate.test_update | AssertionError: Timed out waiting for port! → Intermittent test_direct_update.py TestDirectUpdate.test_update, test_fallback_update.py TestFallbackUpdate.test_update | AssertionError: Timed out waiting for port!
Whiteboard: [mm-osx-106-3]
Summary: Intermittent test_direct_update.py TestDirectUpdate.test_update, test_fallback_update.py TestFallbackUpdate.test_update | AssertionError: Timed out waiting for port! → Intermittent | test_direct_update.py TestDirectUpdate.test_update, test_fallback_update.py TestFallbackUpdate.test_update | AssertionError: Timed out waiting for port!
This might also be a side-effect of bug 1255811.
Depends on: 1255811
My bad. Bug 1255811 was Windows only. So this should be a different issue then.
No longer depends on: 1255811
This is actually an OS X 10.6 only failure. Given that we stopped supporting it I don't feel engaged to get this failure fixed. I would just let it ride the trains down to release and we are set and can close this bug as WFM.
Summary: Intermittent | test_direct_update.py TestDirectUpdate.test_update, test_fallback_update.py TestFallbackUpdate.test_update | AssertionError: Timed out waiting for port! → Intermittent test_direct_update.py TestDirectUpdate.test_update, test_fallback_update.py TestFallbackUpdate.test_update | IOError: Process has been closed (Exit code: 0) (Reason: Timed out waiting for port 2828!)
Summary: Intermittent test_direct_update.py TestDirectUpdate.test_update, test_fallback_update.py TestFallbackUpdate.test_update | IOError: Process has been closed (Exit code: 0) (Reason: Timed out waiting for port 2828!) → Intermittent | test_direct_update.py TestDirectUpdate.test_update, test_fallback_update.py TestFallbackUpdate.test_update | AssertionError: Timed out waiting for port!
OS X 10.6 support basically ended for the 49.0 release. Given that we run our tests without that platform now I don't see a reason to keep this bug open.
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: