Closed
Bug 1143259
Opened 9 years ago
Closed 9 years ago
Don't run tests by default that are failing more than 80 percent of the time
Categories
(Infrastructure & Operations Graveyard :: CIDuty, task)
Tracking
(Not tracked)
RESOLVED
DUPLICATE
of bug 784681
People
(Reporter: jlund, Unassigned)
Details
BUILDERS TOTAL AVERAGE FAILURE BUILDS TIMES % mozilla-inbound_panda_android_test-robocop-2 813 0h,38m,49s 81.80 mozilla-inbound_panda_android_test-remote-tspaint 443 0h,36m,7s 81.26 b2g_try_emulator-l-debug_dep 237 1h,22m,54s 92.41 try_ubuntu32_vm-debug_test-mochitest-jetpack 198 1h,33m,26s 91.92 try_ubuntu64_vm-debug_test-mochitest-jetpack 196 1h,12m,27s 95.41 mozilla-inbound_ubuntu64_vm-debug_test-mochitest-jetpack 168 1h,15m,43s 100.00 try_xp-ix-debug_test-mochitest-jetpack 166 0h,39m,26s 98.19 try_snowleopard-debug_test-mochitest-jetpack 160 0h,15m,21s 99.38 try_panda_android_test-jittest 148 2h,8m,46s 85.81 fx-team_panda_android_test-remote-tspaint 145 0h,36m,20s 84.83 mozilla-inbound_ubuntu64_vm-b2g-emulator_test-marionette 143 0h,34m,12s 100.00 try_win8_64-debug_test-mochitest-jetpack 138 0h,41m,5s 96.38 try_win7-ix-debug_test-mochitest-jetpack 134 0h,39m,46s 99.25 try_ubuntu64_vm-b2g-emulator_test-marionette 133 0h,33m,43s 96.99 mozilla-inbound_xp-ix-debug_test-mochitest-jetpack 118 0h,36m,32s 97.46 mozilla-inbound_snowleopard-debug_test-mochitest-jetpack 114 0h,35m,38s 100.00 mozilla-central_panda_android_test-robocop-2 113 0h,38m,24s 81.42 mozilla-inbound_win7-ix-debug_test-mochitest-jetpack 110 0h,35m,54s 97.27 mozilla-inbound_ubuntu32_vm-debug_test-mochitest-jetpack 110 1h,36m,9s 100.00 mozilla-inbound_win8_64-debug_test-mochitest-jetpack 85 0h,35m,33s 98.82 b2g_b2g-inbound_emulator-l_dep 84 1h,0m,44s 94.05 fx-team_snowleopard-debug_test-mochitest-jetpack 76 0h,35m,5s 100.00 b2g-inbound_ubuntu64_vm-b2g-emulator_test-marionette 76 0h,33m,41s 100.00 fx-team_ubuntu32_vm-debug_test-mochitest-jetpack 75 1h,37m,22s 100.00 fx-team_ubuntu64_vm-debug_test-mochitest-jetpack 74 1h,14m,55s 100.00 fx-team_xp-ix-debug_test-mochitest-jetpack 70 0h,36m,21s 100.00 fx-team_win8_64-debug_test-mochitest-jetpack 70 0h,34m,56s 100.00 try_panda_android_test-remote-tspaint 67 0h,35m,14s 80.60 mozilla-inbound_mountainlion-debug_test-mochitest-jetpack 65 0h,35m,42s 100.00 fx-team_win7-ix-debug_test-mochitest-jetpack 63 0h,36m,14s 100.00 fx-team_mountainlion-debug_test-mochitest-jetpack 62 0h,37m,33s 100.00 while trying to determine where is all our win 7 being used up[1], I came across a number of builders that are run frequently and fail most of the time. The above list represents builders that have run over the last 72 hours, their average job time, and the percentage of failure rate. [1] https://bugzil.la/1142825
Reporter | ||
Comment 1•9 years ago
|
||
I would imagine many of these builders aren't actually considered 'failures' to those who are familiar with the suites as a large majority still of tests within them still pass. However, taking jetpack for example, since they are marked as a failure nearly every time, I would argue that the only time these are actually looked at is when you are explicitly testing jetpack. Therefore, I propose we start disabling some of these by default (I know that's easy for build jobs, can we do it for tests?) or disabling them permanently. removing these from try alone will save us the following over 3 days: try_ubuntu32_vm-debug_test-mochitest-jetpack 198 1h,33m,26s 91.92 <- 326 hours of machine time try_ubuntu64_vm-debug_test-mochitest-jetpack 196 1h,12m,27s 95.41 <- 235 hours try_xp-ix-debug_test-mochitest-jetpack 166 0h,39m,26s 98.19 <- 107 hours try_snowleopard-debug_test-mochitest-jetpack 160 0h,15m,21s 99.38 <- 40 hours try_win8_64-debug_test-mochitest-jetpack 138 0h,41m,5s 96.38 <- 94 hours try_win7-ix-debug_test-mochitest-jetpack 134 0h,39m,46s 99.25 <- 87 hours
Reporter | ||
Comment 2•9 years ago
|
||
mossop, evold: any thoughts wrt to the jetpack builders in comment 1 ^ ? Are they valuable? Are they actually failures?
Flags: needinfo?(evold)
Flags: needinfo?(dtownsend)
Comment 3•9 years ago
|
||
(In reply to Jordan Lund (:jlund) from comment #2) > mossop, evold: any thoughts wrt to the jetpack builders in comment 1 ^ ? Are > they valuable? Are they actually failures? These were only just added, the intention is to fix the leaks that are causing them to fail and then enable them on all branches. I wouldn't object to disabling them by default for now as long as we can run them on try when necessary.
Flags: needinfo?(dtownsend)
Reporter | ||
Comment 4•9 years ago
|
||
thanks mossop for the update. I'll look into turning them off by default
Flags: needinfo?(evold)
Reporter | ||
Comment 5•9 years ago
|
||
mossop pinged me over irc a few days ago and said he has made progress to greening the debug jetpack tests. I expect, in the long run, it is not worth disabling these anywhere.
Comment 6•9 years ago
|
||
(In reply to Jordan Lund (:jlund) from comment #5) > mossop pinged me over irc a few days ago and said he has made progress to > greening the debug jetpack tests. I expect, in the long run, it is not worth > disabling these anywhere. To update this, I just landed bug 1146943 which should fix everything except bug 1144750 which I'm hoping to fix in the next day or two. Then I'll be requesting that the mochitest-jetpack tests are unhidden everywhere.
Updated•9 years ago
|
Summary: tests run by default that are failing more than 80 percent of the time → Don't run tests by default that are failing more than 80 percent of the time
Comment 7•9 years ago
|
||
I filed bug 1149887 for emulator marionette, it's a dead loss. Jetpack is green and unhidden. Android tspaint got shut off, two days before you filed this. robocop-2 appears to have just been retrying a lot, which was fixed the way we always fix Android bustage, moar chunks. You're not going to get anywhere with an overall bug, because every hidden busted job is a special snowflake. You pretty much filed bug 784681 for a new generation. If (and believe me, you don't) you want to know all the jobs that are hidden because they're busted and the ignored bugs about fixing them, and you don't have access to treeherder's Sheriffing menu which is the only reasonable way to see the list of hidden jobs, let me know and I'll give you the whole sad tiresome list. You'll Be Sorry.
Updated•6 years ago
|
Product: Release Engineering → Infrastructure & Operations
Updated•4 years ago
|
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in
before you can comment on or make changes to this bug.
Description
•