Closed
Bug 1285510
Opened 8 years ago
Closed 5 years ago
Intermittent failures of TaskCluster win64 debug builds
Categories
(Infrastructure & Operations Graveyard :: CIDuty, task)
Infrastructure & Operations Graveyard
CIDuty
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: pmoore, Unassigned)
References
Details
See e.g. https://public-artifacts.taskcluster.net/Guj-QMuhTz6N_WBDuPXdvw/0/public/logs/live_backing.log It looks to me like this is due to: Unexpected error: [Error 2] The system cannot find the file specified: 'dist\\crashreporter-symbols\\softokn3.pdb\\6AA8F434C48144C98AE8556CD145563C1\\softokn3.pdb'
|
||
Comment 1•8 years ago
|
||
further up in the log, I see: 6:58:07 INFO - mozmake.EXE[10]: Leaving directory 'z:/task_1467906964/build/src/security/nss/cmd/shlibsign/mangle' 16:58:08 INFO - mozmake.EXE[9]: Entering directory 'z:/task_1467906964/build/src/security/nss/cmd/modutil' 16:58:08 INFO - z:/task_1467906964/build/src/obj-firefox/_virtualenv/Scripts/python.exe z:/task_1467906964/build/src/config/nsinstall.py -t -m 775 z:/task_1467906964/build/src/obj-firefox/security/nss/cmd/modutil/modutil.exe z:/task_1467906964/build/src/obj-firefox/dist/bin 16:58:08 INFO - z:/task_1467906964/build/src/obj-firefox/_virtualenv/Scripts/python.exe z:/task_1467906964/build/src/config/nsinstall.py -t -m 644 z:/task_1467906964/build/src/obj-firefox/security/nss/cmd/modutil/modutil.pdb z:/task_1467906964/build/src/obj-firefox/dist/bin 16:58:08 INFO - mozmake.EXE[9]: Leaving directory 'z:/task_1467906964/build/src/security/nss/cmd/modutil' 16:58:08 INFO - z:/task_1467906964/build/src/config/recurse.mk:71: recipe for target 'config/external/nss/target' failed 16:58:08 INFO - mozmake.EXE[7]: *** [config/external/nss/target] Error 2 16:58:08 INFO - Makefile:33: recipe for target 'gtestxul' failed 16:58:08 INFO - mozmake.EXE[6]: *** [gtestxul] Error 2 16:58:08 INFO - Makefile:21: recipe for target 'gtest' failed 16:58:08 INFO - mozmake.EXE[5]: *** [gtest] Error 2 16:58:08 INFO - z:/task_1467906964/build/src/testing/testsuite-targets.mk:281: recipe for target 'stage-gtest' failed 16:58:08 INFO - mozmake.EXE[4]: *** [stage-gtest] Error 2 16:58:08 INFO - mozmake.EXE[4]: *** Waiting for unfinished jobs.... 16:58:38 INFO - mozmake.EXE[5]: Entering directory 'z:/task_1467906964/build/src/obj-firefox/js/src/tests' and going a bit further up than that, I see: 6:58:06 INFO - mozmake.EXE[10]: Entering directory 'z:/task_1467906964/build/src/security/nss/lib/freebl' 16:58:06 INFO - z:/task_1467906964/build/src/obj-firefox/_virtualenv/Scripts/python.exe z:/task_1467906964/build/src/config/nsinstall.py -t -m 775 z:/task_1467906964/build/src/obj-firefox/security/nss/lib/freebl/freebl3.dll z:/task_1467906964/build/src/obj-firefox/dist/lib 16:58:06 INFO - Traceback (most recent call last): 16:58:06 INFO - File "z:/task_1467906964/build/src/config/nsinstall.py", line 181, in <module> 16:58:06 INFO - sys.exit(_nsinstall_internal(argv[1:])) 16:58:06 INFO - File "z:/task_1467906964/build/src/config/nsinstall.py", line 146, in _nsinstall_internal 16:58:06 INFO - copy_all_entries(args, target) 16:58:06 INFO - File "z:/task_1467906964/build/src/config/nsinstall.py", line 106, in copy_all_entries 16:58:06 INFO - handleTarget(e, dest) 16:58:06 INFO - File "z:/task_1467906964/build/src/config/nsinstall.py", line 132, in handleTarget 16:58:06 INFO - os.remove(targetpath) 16:58:06 INFO - WindowsError: [Error 5] Access is denied: u'z:\\task_1467906964\\build\\src\\obj-firefox\\dist\\lib\\freebl3.dll' 16:58:06 INFO - ../../coreconf/rules.mk:80: recipe for target 'libs' failed 16:58:06 INFO - mozmake.EXE[10]: *** [libs] Error 1 16:58:06 INFO - mozmake.EXE[10]: Leaving directory 'z:/task_1467906964/build/src/security/nss/lib/freebl' 16:58:06 INFO - Makefile:626: recipe for target 'libs' failed 16:58:06 INFO - mozmake.EXE[9]: *** [libs] Error 2 I am not sure if we have an out of space error, or some kind of file permissions error, it seems odd that this would fail where it does.
|
Reporter | |
Comment 2•8 years ago
|
||
(In reply to Joel Maher ( :jmaher ) from comment #1) > I am not sure if we have an out of space error, or some kind of file > permissions error, it seems odd that this would fail where it does. It could be related to https://bugzilla.mozilla.org/show_bug.cgi?id=1285504#c1 - so we should keep an eye out now that that issue is resolved.
|
|
Reporter | |
Comment 3•8 years ago
|
||
I don't think it is related to the disk space issue, e.g. see https://tools.taskcluster.net/task-inspector/#KcQ6qE5rRHW98rIj7be8ng/4 which is a fresh machine with plenty of disk space (I've also logged onto the machine to confirm that there is plenty of disk space available on all drives). A different .pdb file this time: 11:07:48 INFO - Unexpected error: [Error 2] The system cannot find the file specified: 'dist\\crashreporter-symbols\\freebl3.pdb\\16BFA2E2D1E34AEFB7993B55C6CD6CB61\\freebl3.pdb'
|
|
||
Comment 4•8 years ago
|
||
there are a whole list of .pdb files, I learned not too long ago that we have to look much further up in the log to get the original failure- not sure if my first comment was far enough, but I didn't see anything after much scrolling. we end up with the same access denied error further up the log: 11:06:02 INFO - mozmake.EXE[10]: Entering directory 'z:/task_1467971782/build/src/security/nss/lib/freebl' 11:06:02 INFO - z:/task_1467971782/build/src/obj-firefox/_virtualenv/Scripts/python.exe z:/task_1467971782/build/src/config/nsinstall.py -t -m 775 z:/task_1467971782/build/src/obj-firefox/security/nss/lib/freebl/freebl3.dll z:/task_1467971782/build/src/obj-firefox/dist/lib 11:06:02 INFO - z:/task_1467971782/build/src/obj-firefox/_virtualenv/Scripts/python.exe z:/task_1467971782/build/src/config/nsinstall.py -t -m 644 z:/task_1467971782/build/src/obj-firefox/security/nss/lib/freebl/freebl3.pdb z:/task_1467971782/build/src/obj-firefox/dist/lib 11:06:02 INFO - Traceback (most recent call last): 11:06:02 INFO - File "z:/task_1467971782/build/src/config/nsinstall.py", line 181, in <module> 11:06:02 INFO - sys.exit(_nsinstall_internal(argv[1:])) 11:06:02 INFO - File "z:/task_1467971782/build/src/config/nsinstall.py", line 146, in _nsinstall_internal 11:06:02 INFO - copy_all_entries(args, target) 11:06:02 INFO - File "z:/task_1467971782/build/src/config/nsinstall.py", line 106, in copy_all_entries 11:06:02 INFO - handleTarget(e, dest) 11:06:02 INFO - File "z:/task_1467971782/build/src/config/nsinstall.py", line 132, in handleTarget 11:06:02 INFO - os.remove(targetpath) 11:06:02 INFO - WindowsError: [Error 5] Access is denied: u'z:\\task_1467971782\\build\\src\\obj-firefox\\dist\\lib\\freebl3.pdb' 11:06:02 INFO - ../../coreconf/rules.mk:80: recipe for target 'libs' failed 11:06:02 INFO - mozmake.EXE[10]: *** [libs] Error 1 11:06:02 INFO - mozmake.EXE[10]: Leaving directory 'z:/task_1467971782/build/src/security/nss/lib/freebl' 11: I also see this much further up: 11:05:16 INFO - z:/task_1467971782/build/src/obj-firefox/_virtualenv/Scripts/python.exe z:/task_1467971782/build/src/testing/marionette/harness/marionette/tests/print-manifest-dirs.py \ 11:05:16 INFO - z:/task_1467971782/build/src \ 11:05:16 INFO - z:/task_1467971782/build/src/testing/marionette/harness/marionette/tests/unit-tests.ini \ 11:05:16 INFO - | (cd z:/task_1467971782/build/src && xargs tar --exclude=.mkdir.done -chf -) \ 11:05:16 INFO - | (cd dist/test-stage/marionette/tests && tar -xf -) 11:05:16 INFO - tar: toolkit/components/microformats\r: Cannot stat: No such file or directory 11:05:16 INFO - tar: testing/marionette/harness/marionette/tests/unit\r: Cannot stat: No such file or directory 11:05:16 INFO - tar: browser/extensions/loop\r: Cannot stat: No such file or directory 11:05:16 INFO - tar: layout/base/tests/marionette\r: Cannot stat: No such file or directory 11:05:16 INFO - tar: testing/marionette/harness/marionette/tests\r: Cannot stat: No such file or directory 11:05:16 INFO - tar: browser/components/migration/tests/marionette\r: Cannot stat: No such file or directory 11:05:16 INFO - tar: Exiting with failure status due to previous errors 11:05:16 INFO - z:/task_1467971782/build/src/obj-firefox/_virtualenv/Scripts/python.exe z:/task_1467971782/build/src/testing/marionette/harness/marionette/tests/print-manifest-dirs.py \ 11:05:16 INFO - z:/task_1467971782/build/src \ 11:05:16 INFO - z:/task_1467971782/build/src/testing/marionette/harness/marionette/tests/webapi-tests.ini \ 11:05:16 INFO - | (cd z:/task_1467971782/build/src && xargs tar --exclude=.mkdir.done -chf -) \ 11:05:16 INFO - | (cd dist/test-stage/marionette/tests && tar -xf -) 11:05:16 INFO - tar: dom/cellbroadcast/tests/marionette\r: Cannot stat: No such file or directory 11:05:16 INFO - tar: dom/mobilemessage/tests/marionette\r: Cannot stat: No such file or directory 11:05:16 INFO - tar: dom/tethering/tests/marionette\r: Cannot stat: No such file or directory 11:05:16 INFO - tar: dom/icc/tests/marionette\r: Cannot stat: No such file or directory 11:05:16 INFO - tar: dom/system/tests/marionette\r: Cannot stat: No such file or directory 11:05:16 INFO - tar: dom/battery/test/marionette\r: Cannot stat: No such file or directory 11:05:16 INFO - tar: dom/mobileconnection/tests/marionette\r: Cannot stat: No such file or directory 11:05:16 INFO - tar: dom/wifi/test/marionette\r: Cannot stat: No such file or directory 11:05:16 INFO - tar: dom/network/tests/marionette\r: Cannot stat: No such file or directory 11:05:16 INFO - tar: testing/marionette/harness/marionette/tests\r: Cannot stat: No such file or directory 11:05:16 INFO - tar: dom/bluetooth/tests/marionette\r: Cannot stat: No such file or directory 11:05:16 INFO - tar: dom/voicemail/test/marionette\r: Cannot stat: No such file or directory 11:05:16 INFO - tar: dom/events/test/marionette\r: Cannot stat: No such file or directory 11:05:16 INFO - tar: dom/system/gonk/tests/marionette\r: Cannot stat: No such file or directory 11:05:16 INFO - tar: dom/nfc/tests/marionette\r: Cannot stat: No such file or directory 11:05:16 INFO - tar: dom/telephony/test/marionette\r: Cannot stat: No such file or directory 11:05:16 INFO - tar: Exiting with failure status due to previous errors 11:05:16 INFO - mozmake.EXE[4]: Leaving directory 'z:/task_1467971782/build/src/obj-firefox' no idea why these errors are causing the build to fail- we should fix those errors.
|
|
Reporter | |
Comment 5•8 years ago
|
||
Thanks Joel! After analysing further, I did spot we have had *some* successful runs, so it does appear to be intermittent. See https://treeherder.allizom.org/#/jobs?repo=try&filter-platform=windows&filter-job_group_symbol=TC&filter-searchStr=debug%2064 The successful jobs: * https://treeherder.allizom.org/#/jobs?repo=try&revision=4849cd3ca4d20dd4493b1384864d3feab373fe6d&selectedJob=24072859 * https://treeherder.allizom.org/#/jobs?repo=try&revision=6b0da8d64819d0c3eae12b2afd6cda4cb76a9341&selectedJob=24047090 * https://treeherder.allizom.org/#/jobs?repo=try&revision=d7511840f199f29a7895df6e647a63c8dd760977&selectedJob=24045881 * https://treeherder.allizom.org/#/jobs?repo=try&revision=b392bc23489cbc64cfaa81fdebba385138624a51&selectedJob=24032304 * https://treeherder.allizom.org/#/jobs?repo=try&revision=22b87026bb0c8ac74ff41f69fdf346f24c14c5e7&selectedJob=24029620 However, most of the other jobs failed... But it *does* seem to be intermittent.
|
|
Reporter | |
Updated•8 years ago
|
Summary: TaskCluster win64 debug builds are failing on try → Intermittent failures of TaskCluster win64 debug builds on try
|
|
Reporter | |
Comment 6•8 years ago
|
||
Since all trees exhibit this, renaming bug... For example try: https://treeherder.allizom.org/#/jobs?repo=try&filter-platform=windows&filter-job_group_symbol=TC&filter-searchStr=debug%2064 mozilla inbound: https://treeherder.allizom.org/#/jobs?repo=mozilla-inbound&filter-platform=windows&filter-job_group_symbol=TC&filter-searchStr=debug%2064 mozilla central: https://treeherder.allizom.org/#/jobs?repo=mozilla-central&filter-platform=windows&filter-job_group_symbol=TC&filter-searchStr=debug%2064
Summary: Intermittent failures of TaskCluster win64 debug builds on try → Intermittent failures of TaskCluster win64 debug builds
|
||
Comment 7•8 years ago
|
||
i believe this is caused by a failure to delete files (because some other process still has the file handle open). see bug 1287258
Depends on: 1287258
|
Assignee | |
Updated•6 years ago
|
Component: Platform Support → Buildduty
Product: Release Engineering → Infrastructure & Operations
|
||
Comment 8•5 years ago
|
||
Rob, Can you please take a look ? Can this bug be closed ?
Flags: needinfo?(rthijssen)
|
|
||
Comment 9•5 years ago
|
||
yes, this was dealt with some time ago and was caused by temp data delete failures.
Status: NEW → RESOLVED
Closed: 5 years ago
Flags: needinfo?(rthijssen)
Resolution: --- → FIXED
|
||
Updated•4 years ago
|
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in
before you can comment on or make changes to this bug.
Description
•