Closed Bug 1143259 Opened 9 years ago Closed 9 years ago

Don't run tests by default that are failing more than 80 percent of the time

Categories

(Infrastructure & Operations Graveyard :: CIDuty, task)

x86
macOS
task
Not set
normal

Tracking

(Not tracked)

RESOLVED DUPLICATE of bug 784681

People

(Reporter: jlund, Unassigned)

Details

BUILDERS                                                  TOTAL   AVERAGE FAILURE
                                                          BUILDS   TIMES     %
mozilla-inbound_panda_android_test-robocop-2               813  0h,38m,49s 81.80 
mozilla-inbound_panda_android_test-remote-tspaint          443  0h,36m,7s  81.26 
b2g_try_emulator-l-debug_dep                               237  1h,22m,54s 92.41 
try_ubuntu32_vm-debug_test-mochitest-jetpack               198  1h,33m,26s 91.92 
try_ubuntu64_vm-debug_test-mochitest-jetpack               196  1h,12m,27s 95.41 
mozilla-inbound_ubuntu64_vm-debug_test-mochitest-jetpack   168  1h,15m,43s 100.00
try_xp-ix-debug_test-mochitest-jetpack                     166  0h,39m,26s 98.19 
try_snowleopard-debug_test-mochitest-jetpack               160  0h,15m,21s 99.38 
try_panda_android_test-jittest                             148  2h,8m,46s  85.81 
fx-team_panda_android_test-remote-tspaint                  145  0h,36m,20s 84.83 
mozilla-inbound_ubuntu64_vm-b2g-emulator_test-marionette   143  0h,34m,12s 100.00
try_win8_64-debug_test-mochitest-jetpack                   138  0h,41m,5s  96.38 
try_win7-ix-debug_test-mochitest-jetpack                   134  0h,39m,46s 99.25 
try_ubuntu64_vm-b2g-emulator_test-marionette               133  0h,33m,43s 96.99 
mozilla-inbound_xp-ix-debug_test-mochitest-jetpack         118  0h,36m,32s 97.46 
mozilla-inbound_snowleopard-debug_test-mochitest-jetpack   114  0h,35m,38s 100.00
mozilla-central_panda_android_test-robocop-2               113  0h,38m,24s 81.42 
mozilla-inbound_win7-ix-debug_test-mochitest-jetpack       110  0h,35m,54s 97.27 
mozilla-inbound_ubuntu32_vm-debug_test-mochitest-jetpack   110  1h,36m,9s  100.00
mozilla-inbound_win8_64-debug_test-mochitest-jetpack       85   0h,35m,33s 98.82 
b2g_b2g-inbound_emulator-l_dep                             84   1h,0m,44s  94.05 
fx-team_snowleopard-debug_test-mochitest-jetpack           76   0h,35m,5s  100.00
b2g-inbound_ubuntu64_vm-b2g-emulator_test-marionette       76   0h,33m,41s 100.00
fx-team_ubuntu32_vm-debug_test-mochitest-jetpack           75   1h,37m,22s 100.00
fx-team_ubuntu64_vm-debug_test-mochitest-jetpack           74   1h,14m,55s 100.00
fx-team_xp-ix-debug_test-mochitest-jetpack                 70   0h,36m,21s 100.00
fx-team_win8_64-debug_test-mochitest-jetpack               70   0h,34m,56s 100.00
try_panda_android_test-remote-tspaint                      67   0h,35m,14s 80.60 
mozilla-inbound_mountainlion-debug_test-mochitest-jetpack  65   0h,35m,42s 100.00
fx-team_win7-ix-debug_test-mochitest-jetpack               63   0h,36m,14s 100.00
fx-team_mountainlion-debug_test-mochitest-jetpack          62   0h,37m,33s 100.00


while trying to determine where is all our win 7 being used up[1], I came across a number of builders that are run frequently and fail most of the time. The above list represents builders that have run over the last 72 hours, their average job time, and the percentage of failure rate.


[1]  https://bugzil.la/1142825
I would imagine many of these builders aren't actually considered 'failures' to those who are familiar with the suites as a large majority still of tests within them still pass.

However, taking jetpack for example, since they are marked as a failure nearly every time, I would argue that the only time these are actually looked at is when you are explicitly testing jetpack. Therefore, I propose we start disabling some of these by default (I know that's easy for build jobs, can we do it for tests?) or disabling them permanently.

removing these from try alone will save us the following over 3 days:
try_ubuntu32_vm-debug_test-mochitest-jetpack    198  1h,33m,26s 91.92 <- 326 hours of machine time
try_ubuntu64_vm-debug_test-mochitest-jetpack    196  1h,12m,27s 95.41 <- 235 hours
try_xp-ix-debug_test-mochitest-jetpack          166  0h,39m,26s 98.19 <- 107 hours
try_snowleopard-debug_test-mochitest-jetpack    160  0h,15m,21s 99.38 <- 40 hours
try_win8_64-debug_test-mochitest-jetpack        138  0h,41m,5s  96.38 <- 94 hours
try_win7-ix-debug_test-mochitest-jetpack        134  0h,39m,46s 99.25 <- 87 hours
mossop, evold: any thoughts wrt to the jetpack builders in comment 1 ^ ? Are they valuable? Are they actually failures?
Flags: needinfo?(evold)
Flags: needinfo?(dtownsend)
(In reply to Jordan Lund (:jlund) from comment #2)
> mossop, evold: any thoughts wrt to the jetpack builders in comment 1 ^ ? Are
> they valuable? Are they actually failures?

These were only just added, the intention is to fix the leaks that are causing them to fail and then enable them on all branches. I wouldn't object to disabling them by default for now as long as we can run them on try when necessary.
Flags: needinfo?(dtownsend)
thanks mossop for the update. I'll look into turning them off by default
Flags: needinfo?(evold)
mossop pinged me over irc a few days ago and said he has made progress to greening the debug jetpack tests. I expect, in the long run, it is not worth disabling these anywhere.
(In reply to Jordan Lund (:jlund) from comment #5)
> mossop pinged me over irc a few days ago and said he has made progress to
> greening the debug jetpack tests. I expect, in the long run, it is not worth
> disabling these anywhere.

To update this, I just landed bug 1146943 which should fix everything except bug 1144750 which I'm hoping to fix in the next day or two. Then I'll be requesting that the mochitest-jetpack tests are unhidden everywhere.
Summary: tests run by default that are failing more than 80 percent of the time → Don't run tests by default that are failing more than 80 percent of the time
Depends on: 1149887
I filed bug 1149887 for emulator marionette, it's a dead loss. Jetpack is green and unhidden. Android tspaint got shut off, two days before you filed this. robocop-2 appears to have just been retrying a lot, which was fixed the way we always fix Android bustage, moar chunks.

You're not going to get anywhere with an overall bug, because every hidden busted job is a special snowflake. You pretty much filed bug 784681 for a new generation. If (and believe me, you don't) you want to know all the jobs that are hidden because they're busted and the ignored bugs about fixing them, and you don't have access to treeherder's Sheriffing menu which is the only reasonable way to see the list of hidden jobs, let me know and I'll give you the whole sad tiresome list. You'll Be Sorry.
Status: NEW → RESOLVED
Closed: 9 years ago
No longer depends on: 1149887
Resolution: --- → DUPLICATE
Product: Release Engineering → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.