Closed Bug 1361792 Opened 7 years ago Closed 6 years ago

(Firefox Screenshots) 2.11 - 48.22% most all tests (linux64, osx-10-10, windows7-32, windows8-64) regression on push 8090b323f5d1 (Wed May 3 2017)

Categories

(Firefox :: General, defect)

Product:
Firefox
Component:
General
Version:
55 Branch
Type:
defect
Priority:
Not set
Severity:
major

Tracking

()

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox-esr45 --- unaffected
firefox-esr52 --- unaffected
firefox53 --- unaffected
firefox54 --- unaffected
firefox55 + fixed

People

(Reporter: jmaher, Unassigned)

References

Details

(4 keywords, Whiteboard: [MemShrink:P1]) 

Talos has detected a Firefox performance regression from push 8090b323f5d11719ebb83c99b3b0de26e69234e8. As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

 48%  tsvg_static summary linux64 pgo      56.3 -> 83.45
 48%  tabpaint summary osx-10-10 opt       86.16 -> 127.43
 48%  tabpaint summary windows7-32 opt     84.38 -> 124.71
 44%  tsvg_static summary osx-10-10 opt    66.64 -> 95.69
 43%  tabpaint summary linux64 pgo         67.69 -> 96.63
 42%  tsvg_static summary linux64 opt      76.68 -> 109.03
 41%  tsvg_static summary windows8-64 opt  71.17 -> 100.57
 38%  tsvg_static summary windows7-32 opt  78.66 -> 108.26
 37%  tp5o responsiveness linux64 opt e10s 4.87 -> 6.68
 37%  tabpaint summary linux64 opt         86.53 -> 118.52
 33%  tp5o responsiveness linux64 pgo e10s 3.97 -> 5.29
 33%  tabpaint summary windows8-64 opt     94.19 -> 125.25
 20%  tps summary linux64 pgo              38.04 -> 45.56
 20%  tart summary windows8-64 opt         6.43 -> 7.7
 17%  tps summary linux64 opt              46.27 -> 54.13
 17%  tps summary windows7-32 opt          48.52 -> 56.57
 16%  tpaint linux64 pgo                   215.57 -> 250.15
 15%  tart summary linux64 pgo             4.34 -> 4.98
 15%  tart summary linux64 opt             5.98 -> 6.87
 14%  tart summary osx-10-10 opt           10.82 -> 12.3
 13%  tp5o summary osx-10-10 opt           283.14 -> 320.19
 12%  tart summary linux64 pgo e10s        4.77 -> 5.36
 12%  tp5o summary linux64 pgo             237.02 -> 265.57
 12%  tpaint linux64 opt                   276.39 -> 308.64
 12%  tsvg_static summary linux64 opt e10s 89.28 -> 99.68
 11%  tp5o summary linux64 opt             335.27 -> 373.15
 11%  tp5o summary windows8-64 opt         328.79 -> 365.23
 11%  tpaint osx-10-10 opt                 374.12 -> 415.23
 11%  tpaint windows7-32 opt               297.37 -> 329.79
 11%  tpaint windows8-64 opt               286.28 -> 316.84
 10%  tart summary linux64 opt e10s        6.63 -> 7.27
 10%  tart summary windows7-32 opt         7.04 -> 7.71
  9%  tsvgx summary windows8-64 opt        283.21 -> 308.45
  9%  tp5o summary windows7-32 opt         360.45 -> 391.71
  8%  tp5o Main_RSS linux64 pgo e10s       165793106.12 -> 178737390.88
  8%  tp5o Main_RSS linux64 opt e10s       168489633.04 -> 181199316.66
  7%  tp5o summary linux64 pgo e10s        269.17 -> 288.03
  7%  tsvgr_opacity summary linux64 pgo    361.33 -> 386.29
  7%  tsvgr_opacity summary windows8-64 opt 373.41 -> 399.06
  7%  tsvgr_opacity summary osx-10-10 opt  419 -> 447.18
  7%  tsvgx summary linux64 pgo            345.87 -> 368.85
  6%  cart summary linux64 pgo             21.17 -> 22.45
  6%  tsvgx summary linux64 opt            488.21 -> 517.54
  6%  tsvgx summary osx-10-10 opt          460.11 -> 487.47
  6%  tp5o summary linux64 opt e10s        372.95 -> 394.05
  6%  tsvgr_opacity summary linux64 opt    491.73 -> 519.17
  5%  a11yr summary linux64 pgo            351.89 -> 371.09
  5%  tp5o Private Bytes windows7-32 opt   211515317.54 -> 222509787.18
  5%  tsvgr_opacity summary windows7-32 opt 556.82 -> 584.31
  5%  tsvgx summary windows7-32 opt        557.47 -> 582.62
  4%  tp5o Main_RSS linux64 opt            242549554.56 -> 253072708.75
  4%  tp5o Main_RSS windows7-32 opt        177225548.77 -> 184867892.33
  4%  damp summary linux64 pgo e10s        236.74 -> 246.6
  4%  tp5o Main_RSS linux64 pgo            246938981.38 -> 256341428.03
  4%  damp summary linux64 opt e10s        291.21 -> 301.9
  4%  cart summary linux64 opt             28.21 -> 29.22
  3%  damp summary linux64 opt             304.15 -> 313.55
  3%  cart summary linux64 pgo e10s        22.56 -> 23.2
  3%  tp5o Main_RSS osx-10-10 opt          396221693.86 -> 407116924.66
  2%  tsvgr_opacity summary linux64 pgo e10s332.79 -> 340.72
  2%  cart summary linux64 opt e10s        29.73 -> 30.36


You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=6329

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the Talos jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://wiki.mozilla.org/Buildbot/Talos/Tests

For information on reproducing and debugging the regression, either on try or locally, see: https://wiki.mozilla.org/Buildbot/Talos/Running

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Our wiki page outlines the common responses and expectations: https://wiki.mozilla.org/Buildbot/Talos/RegressionBugsHandling
Ian, did screenshots change anything major on the 6.3.0 -> 6.6.0? When we ran talos previously, I'm sure it was OK...
Flags: needinfo?(ianb)
Changelog here: https://github.com/mozilla-services/screenshots/blob/master/CHANGELOG.md

Here are the things that seem like they could potentially have an affect:

- Force onboarding when you visit /#hello. Fixes #2643 48b37fe
- manually dim toolbar button when disabledOn Windows and Linux, WebExtension toolbar buttons are not automatically dimmed when a button is disabled (bug 1204609). Fixes #2708 b1415f6
- Shutdown the embedded webExtension when bootstrap is asked to shutdown (#2712) 9a23339
- Stop Errors being shown when browser.tabs.get() fails as a result of tabs going away too quick, e.g. browser mochitests. 11ec080
- Remove embedded web extension in install manifest. (#2688). This causes the embedded web extension to be parsed at startup, and triggers a race condition in existing Firefox code during startup on a clean profile between AddonManager and devtools code. Removing this is not an issue for Screenshots because it delays startup of the embedded web extension until "sessionstore-windows-restored" is observed anyway. See https://bugzilla.mozilla.org/show_bug.cgi?id=1356394 for more info. 2fa25c6

These are the changes I can see that affect code that runs even when Screenshots isn't active.
Flags: needinfo?(ianb)
We discussed this issue a bit in #perf just now. Screenshots currently waits for the sessionstore-windows-restored event before starting up the WebExtension code; :kmag suggested removing the delay and re-running the Talos tests.

I've done that and pushed to Try: https://treeherder.mozilla.org/#/jobs?repo=try&revision=2e54ba71d896d77ba28e7ac1a26d3c594b622254

Once results are in, I'll work with jmaher and kmag to figure out next steps.
Kris also suggested that we try removing the code that toggles toolbar button state when tabs are activated. In particular, we are toggling the button icon with each tab switch, which isn't correct.

I've removed the onActivated listener, and applied that change on top of the startup delay change, and pushed to Try with a 5x Talos run: https://treeherder.mozilla.org/#/jobs?repo=try&revision=e3b63a182431c67261dde950942df15d22acb241

Current Screenshots master has moved ahead of the 6.6.0 release, so, to avoid losing track of the sequence of patches, I've pushed the branch here: https://github.com/mozilla-services/screenshots/tree/perf-investigation-6.6.0
A regression of this scale on this many Talos tests makes this extremely difficult for others to evaluate the performance impacts of their code.  While we're investigating the cause here we should probably back out bug 1356243 from inbound while the investigations here go on.  We should really try doing that before this gets merged around the regressions start to spread around to m-c and other integration branches...

Mark, do you mind doing that please?
Flags: needinfo?(standard8)
Used the wrong Try syntax in the last two builds. Trying again:

Screenshots 6.6.0 + Remove late startup:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=db5b3765aa636d8468f21e92f3acddb6da5f01ae

Screenshots 6.6.0 + Remove late startup + remove tabs.onActivated:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=80d0cdf6164b66df353115ffb9b8c27e1f76b329
I've backed out bug 1356243 so I can get an inbound merge in.
Added some additional runs to try to pin down where the regression was introduced:

Screenshots 6.5.0 + Remove late startup:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=930690ab98edf5b0bc6269f59a6d4700d1b85d86

Screenshots 6.5.0:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=5fa2e58e3b4fe7301235bca30c9f86a1e257c88d

I have some additional changes to push, but looks like the Try server is currently closed.

Also clearing ni? for Standard8, as the patch has been backed out.
Flags: needinfo?(standard8)
Component: Untriaged → General
also areweslimyet found a bunch of regressions:
== Change summary for alert #6342 (as of May 03 2017 08:11 UTC) ==

Regressions:

 17%  JS summary linux32 opt      117920061.38 -> 138013916.84
 16%  JS summary windows7-32-vm opt 123519999.94 -> 143796520.52
 15%  JS summary linux64 opt      168628624.87 -> 194704840.22
 15%  JS summary windows10-64-vm opt 174414525.01 -> 199790609.46
 12%  Explicit Memory summary linux32 opt 114699970.57 -> 128108881.37
 11%  Explicit Memory summary linux64 opt 152470470.51 -> 169743370.94
 11%  Explicit Memory summary windows7-32-vm opt 113754127.46 -> 125804769.5
  9%  Explicit Memory summary windows10-64-vm opt 150836953.8 -> 164350338.56
  8%  Resident Memory summary windows7-32-vm opt 406758961.61 -> 440093651.94
  8%  Resident Memory summary windows10-64-vm opt 611749758.37 -> 659143509.34
  6%  Heap Unclassified summary linux64 opt 51654024.14 -> 54853485.17
  6%  Heap Unclassified summary windows7-32-vm opt 36022882.61 -> 38074312.6
  5%  Heap Unclassified summary windows10-64-vm opt 41109463.08 -> 43011480.4

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=6342
and the backout has a bunch of improvements:
https://treeherder.mozilla.org/perf.html#/alerts?id=6352
Trying again, changes applied on top of this morning’s m-c, with the enabling pref change cherry-picked in:

6.6.0 release enabled:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=a02722f542d2758f9ddf9f7de8396c5622449ee8

6.6.0 release enabled + startup delay removed:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=0e23678adb4f5937121392c73103d366874afe4f

6.6.0 release enabled + onActivated removed + startup delay removed:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=a6dbd4efb3237bb67cf58014c251eb2bbcef65dc

6.6.0 release enabled + all tab code disabled + onActivated removed + startup delay removed:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=87e9b73df70102ce3582a09b7bc60e7b265faad5
One more Try build in the queue:

Current Screenshots master (6.6.0 + lots of changes) + all three perf patches:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=ca8b8b539cbf0fbc372daac3d4b8365899fb0fdf
Assignee: nobody → jhirsch
Status: NEW → ASSIGNED
Well, it looks like with just the startup delay and onActivated listeners removed, the only serious regressions are in sessionrestore and ts_paint:

https://treeherder.mozilla.org/perf.html#/compare?originalProject=mozilla-central&originalRevision=a8d597ee6dd5&newProject=try&newRevision=87e9b73df70102ce3582a09b7bc60e7b265faad5&framework=1&filter=windows7&showOnlyImportant=0

That's a bit unfortunate, but I've already paid for at least a 10% regression in both of those for the sake of being able to run WebExtensions as system add-ons, and once bug 1359653, bug 1361900, bug 1356826, and bug 1358846 land, we'll still be significantly ahead.

Still, it would be nice to do some profiling to find out exactly where the extra overhead is coming from.
Some other things in the sessionrestore profile (https://perfht.ml/2peilLd):

- extensionURILoadableByAnyone is taking at least 9ms. This is bug 1322235. I've been planning to move this to C++, which should help a lot.

- Creating the CustomizeableUI widget takes at least 5ms.

- We spend about 4ms in i18n.getMessage. That can probably be made a bit faster, but it would be nice if those calls happened lazily. I don't think any of them are actually used by the time sessionrestore is finished.

- We spend something like 20ms in binding code, mainly generating lazy bindings and loading the lazy APIs those bindings require. That can probably be improved a bit, but I'm not sure how much. Avoiding calling APIs before absolutely necessary would be preferable, though.

- A bunch of the rest of the time is spent just loading code, which bug 1359653 and bug 1361900 should help with.

OOP extensions will help with a lot of this, since a huge chunk of this happens in the extension process, which happens to be the parent process at the moment.
Depends on: 1362225
Depends on: 1362226
Depends on: 1359653
Depends on: 1322235
I've landed the Screenshots talos fixes in bug 1362234.

Mark has relanded the patch that turns on Screenshots:
https://hg.mozilla.org/integration/mozilla-inbound/rev/caa721034f47f4bc201d018f2f6659fd2bc839b3

:jmaher, if the Talos run for Mark's push looks good, are we good to close this bug? Or do we want to leave this open until the dependent webextension bugs are fixed?
Assignee: jhirsch → nobody
Status: ASSIGNED → NEW
Flags: needinfo?(jmaher)
the push I see has many of the same regressions, some are not there or are smaller scale- it is hard to tell until we get more data.

Using the data and reasoning we had before, I would advocate for backing this out.  I know this is on the radar and has more specific action items; :ehsan, what thoughts do you have?
Flags: needinfo?(jmaher) → needinfo?(ehsan)
(In reply to Joel Maher ( :jmaher) from comment #16)
> Using the data and reasoning we had before, I would advocate for backing
> this out.  I know this is on the radar and has more specific action items;
> :ehsan, what thoughts do you have?

Can you link to the new regression data in a form similar to comment 0 please?  Is the issue that the regressions don't show up on the try server but only show up on inbound?  I'm trying to understand how Jared's trial runs on the try server here didn't show any regressions until Mark relanded.  :-/

But at any rate, if this is still regressing many Talos tests across the board like before it should be backed out for the same reason.  But if there is a problem that prevents us from detecting the regression on the try server it would be helpful to flag it out, I hate to keep asking people to back this out repeatedly.  :-(
Flags: needinfo?(ehsan)
(And FWIW this time the code got landed and merged to m-c so the problem that I was trying to prevent the last time has probably occurred and it's a bit too late...)
here is a view on compare:
https://treeherder.mozilla.org/perf.html#/compare?originalProject=mozilla-inbound&originalRevision=a4b157d9d0e7&newProject=mozilla-inbound&newRevision=caa721034f47f4bc201d018f2f6659fd2bc839b3&framework=1

it will take an hour or so of focus time (maybe tomorrow) where I can get the alerts into the right summary and post in the bug.
(In reply to Joel Maher ( :jmaher) from comment #19)
> here is a view on compare:
> https://treeherder.mozilla.org/perf.html#/compare?originalProject=mozilla-
> inbound&originalRevision=a4b157d9d0e7&newProject=mozilla-
> inbound&newRevision=caa721034f47f4bc201d018f2f6659fd2bc839b3&framework=1
> 
> it will take an hour or so of focus time (maybe tomorrow) where I can get
> the alerts into the right summary and post in the bug.

Thanks Joel.  Skimming the results, this looks basically the same to me in that they're happening all across the board.  Some of the regressions have improved a bit, but still, unfortunately I still we should back out bug 1356243 again.  :/
A lot of them should be improved by bug 1362224, which will be landing in a few minutes, and bug 1362550, which should land today.

The ts_paint and sessionrestore regressions are expected. The non-main file IO regressions only look bad because they happen to be non-main rather than main file IO.
and the next landing is:
== Change summary for alert #6418 (as of May 05 2017 16:04 UTC) ==

Regressions:

1328%  tp5n nonmain_startup_fileio windows7-32 opt      42717.88 -> 609992.58
 43%  tsvg_static summary osx-10-10 opt                66.7 -> 95.64
 42%  tsvg_static summary windows8-64 opt              70.59 -> 99.96
 39%  tabpaint summary osx-10-10 opt                   85.29 -> 118.71
 39%  tsvg_static summary windows7-32 opt              77.71 -> 107.97
 38%  tsvg_static summary linux64 opt                  78.61 -> 108.09
 35%  tabpaint summary windows7-32 opt                 84.98 -> 114.84
 24%  tabpaint summary windows8-64 opt                 93.94 -> 116.79
 24%  tabpaint summary linux64 opt                     85.69 -> 106.19
 23%  tsvg_static summary osx-10-10 opt e10s           66.11 -> 81.23
 15%  sessionrestore_no_auto_restore windows7-32 pgo e10s686.08 -> 789.67
 15%  tsvg_static summary windows8-64 opt e10s         61.24 -> 70.28
 14%  tpaint linux64 opt                               259.26 -> 296.39
 12%  tabpaint summary osx-10-10 opt e10s              54.52 -> 61.31
 12%  tp5o summary osx-10-10 opt                       285.54 -> 320.25
 12%  tpaint osx-10-10 opt                             362.96 -> 406.45
 12%  sessionrestore windows7-32 pgo e10s              682.92 -> 764.42
 11%  sessionrestore_no_auto_restore windows7-32 opt   901.5 -> 1001.25
 11%  sessionrestore osx-10-10 opt e10s                903.19 -> 1003
 11%  tp5o summary windows8-64 opt                     330.1 -> 365.8
 11%  sessionrestore windows8-64 opt                   808.58 -> 895.67
 11%  sessionrestore windows7-32 opt                   876.25 -> 970.17
 11%  tpaint windows7-32 opt                           287.69 -> 318.48
 11%  sessionrestore_no_auto_restore windows8-64 opt   830.17 -> 918.75
 10%  sessionrestore windows8-64 opt e10s              762.42 -> 841.67
 10%  tpaint windows8-64 opt                           276.7 -> 304.52
 10%  sessionrestore_no_auto_restore windows8-64 opt e10s789.38 -> 864.83
  9%  tp5o summary osx-10-10 opt e10s                  284.71 -> 310.64
  9%  sessionrestore_no_auto_restore osx-10-10 opt e10s932.54 -> 1015.5
  9%  tps summary osx-10-10 opt                        48.64 -> 52.94
  8%  tart summary windows8-64 opt                     6.43 -> 6.97
  8%  sessionrestore osx-10-10 opt                     995 -> 1075.83
  8%  tart summary windows8-64 opt e10s                6.09 -> 6.58
  8%  sessionrestore_no_auto_restore osx-10-10 opt     1020.21 -> 1099.58
  8%  tsvgx summary windows8-64 opt                    285.36 -> 307.48
  7%  tart summary osx-10-10 opt                       10.7 -> 11.48
  7%  tart summary osx-10-10 opt e10s                  10.68 -> 11.45
  7%  tp5o Main_RSS linux64 opt e10s                   168198338.47 -> 179758308.98
  7%  tp5o summary windows8-64 opt e10s                310.93 -> 332
  6%  tps summary windows8-64 opt                      46.09 -> 49
  6%  tsvgr_opacity summary windows8-64 opt            373.53 -> 397.05
  6%  tp5o Main_RSS osx-10-10 opt e10s                 336580012.54 -> 355441383.12
  6%  tsvgr_opacity summary osx-10-10 opt              424.49 -> 448.25
  6%  ts_paint windows8-64 opt                         920.42 -> 971.25
  5%  ts_paint windows7-32 opt                         1022.25 -> 1078.25
  5%  tsvgr_opacity summary windows7-32 pgo            427.97 -> 451.16
  5%  tsvgx summary osx-10-10 opt                      462.08 -> 486.97
  5%  tsvgr_opacity summary windows7-32 opt            559.11 -> 589.06
  5%  damp summary windows7-32 opt e10s                286.13 -> 300.87
  5%  tpaint osx-10-10 opt e10s                        301.31 -> 316.41
  5%  ts_paint windows8-64 opt e10s                    880.5 -> 921.17
  4%  tsvgx summary windows7-32 opt                    557.29 -> 581.93
  4%  tsvgr_opacity summary osx-10-10 opt e10s         343.88 -> 358.53
  4%  tp5o Main_RSS linux64 opt                        242287740.66 -> 252505143.37
  4%  tabpaint summary windows8-64 opt e10s            52.61 -> 54.72
  4%  tp5o Private Bytes windows7-32 opt               211895573.16 -> 220319443.03
  4%  ts_paint osx-10-10 opt e10s                      1159.19 -> 1205.08
  4%  tpaint windows7-32 pgo e10s                      211.38 -> 219.6
  4%  damp summary windows7-32 pgo e10s                231.15 -> 240.12
  4%  tsvgx summary windows7-32 pgo                    496.99 -> 515.39
  4%  tp5o Main_RSS windows7-32 opt                    176469456.38 -> 182974035.35
  4%  ts_paint windows7-32 pgo e10s                    863.58 -> 895.33
  4%  damp summary windows8-64 opt                     305.26 -> 316.26
  3%  ts_paint osx-10-10 opt                           1140.54 -> 1179.17
  3%  damp summary windows8-64 opt e10s                262.37 -> 271.23
  3%  tsvgr_opacity summary windows8-64 opt e10s       315.61 -> 326.13
  3%  tp5o Main_RSS osx-10-10 opt                      395190327.77 -> 406063857.29
  3%  cart summary windows8-64 opt                     27.84 -> 28.59

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=6418

this was backed out again- I will post a reference to the improvements when they are in.
(In reply to Kris Maglione [:kmag] (busy; behind on reviews) from comment #21)
> The ts_paint and sessionrestore regressions are expected.

Expected as in you were expecting for them to happen or as in OK for them to happen?
(In reply to :Ehsan Akhgari (super long backlog, slow to respond) from comment #23)
> (In reply to Kris Maglione [:kmag] (busy; behind on reviews) from comment
> #21)
> > The ts_paint and sessionrestore regressions are expected.
> 
> Expected as in you were expecting for them to happen or as in OK for them to
> happen?

Both. I've been working on improving the load time of the WebExtensions framework as much as possible, but there's no getting around the fact that loading it means loading and initializing a lot of extra code. So since some regression is unavoidable there, I've been working on improving the startup performance in other areas (particularly the add-on manager and script loader) so we still come out ahead even after we load the framework for all users.

The improvements that I've made so far already make up for the regressions this introduces in those tests, and there are more coming.
(In reply to Kris Maglione [:kmag] (busy; behind on reviews) from comment #24)
> (In reply to :Ehsan Akhgari (super long backlog, slow to respond) from
> comment #23)
> > (In reply to Kris Maglione [:kmag] (busy; behind on reviews) from comment
> > #21)
> > > The ts_paint and sessionrestore regressions are expected.
> > 
> > Expected as in you were expecting for them to happen or as in OK for them to
> > happen?
> 
> Both. I've been working on improving the load time of the WebExtensions
> framework as much as possible, but there's no getting around the fact that
> loading it means loading and initializing a lot of extra code.

That's certainly true.

> So since some
> regression is unavoidable there, I've been working on improving the startup
> performance in other areas (particularly the add-on manager and script
> loader) so we still come out ahead even after we load the framework for all
> users.
> 
> The improvements that I've made so far already make up for the regressions
> this introduces in those tests, and there are more coming.

Well, wait.  Implementing the screenshots feature as a WebExtension is a choice though, isn't it?  What I would love to see here is to see the screenshots feature shipped alongside all of the improvements that you have been working on, not have that feature roll back all of those improvements.  I guess if the latter happens then maybe technically we haven't regressed something but that seems like a weird thing to do when we also have the option of shipping this feature without incurring these regressions.  Am I missing something in the trade-off?
(In reply to :Ehsan Akhgari (super long backlog, slow to respond) from comment #25)
> > So since some
> > regression is unavoidable there, I've been working on improving the startup
> > performance in other areas (particularly the add-on manager and script
> > loader) so we still come out ahead even after we load the framework for all
> > users.
> >
> > The improvements that I've made so far already make up for the regressions
> > this introduces in those tests, and there are more coming.
>
> Well, wait.  Implementing the screenshots feature as a WebExtension is a
> choice though, isn't it?

To some extent, yes. But not really. It's going to need to be a WebExtension
at some point soon, so implementing it some other way means wasted engineering
time that we can't afford.

And the overhead we're talking about here is for loading the framework, not
for loading a particular extension. Most users have some extension installed,
if only an anti-virus or some other piece of side-loaded crapware, and we're
moving toward a world where most extensions are WebExtensions, so most users
are paying this cost anyway. Having some WebExtension enabled for talos tests
gives us a better picture of what most of our users are experiencing.

> What I would love to see here is to see the screenshots feature shipped
> alongside all of the improvements that you have been working on, not have
> that feature roll back all of those improvements. I guess if the latter
> happens then maybe technically we haven't regressed something but that seems
> like a weird thing to do when we also have the option of shipping this
> feature without incurring these regressions.  Am I missing something in the
> trade-off?

That idea had occurred to me, but it seems pointlessly political. I don't have
to time or energy at this point to leave performance improvements bit rotting
in my patch queue just so that they can land at the same time as some other
changeset.

As far as I'm concerned, all that matters is that we ship 55 with better net
performance than 54, and that the WebExtensions project has a net positive
impact on it. My job at the moment is to make sure that we ship a good
WebExtensions product, so most of the work I do needs to contribute to that in
some way. Right now, part of that means improving startup performance enough
that we can ship new framework code by default without hurting performance,
which means that I get to work on platform changes that put us in a much
better place than we were before we started shipping that framework.

But I have a *lot* of work to do at this point, and I'm having enough trouble
getting patches shipped without trying to coordinate them with other
changesets for some arbitrary impact on talos runs. So if I have to play those
sorts of political games, it probably means we go on not shipping any
WebExtensions by default, and pretending that the performance impact of the
framework doesn't matter, and I move onto other things.
That said, if you want the improvements to ship at the same time as the regressions from this patch, will these do?

https://treeherder.mozilla.org/perf.html#/alerts?id=6440
https://treeherder.mozilla.org/perf.html#/alerts?id=6439
what about the tart, svg, damp, tpaint, tp5, tabpaint, and memory regressions?  What usually happens on large regressions is we fix 80% of them and ignore the rest.  If we let just ts_paint and sessionrestore stick until a series of bugs were fixed that wouldn't be so bad.

I am looking at the above alerts to see if they are related to the backout of screenshots or the changes made- will organize then when more data comes in!
(In reply to Joel Maher ( :jmaher) from comment #28)
> what about the tart, svg, damp, tpaint, tp5, tabpaint, and memory
> regressions?

Most of the others are caused by an unusually expensive tab listener, and should be fixed by bug 1362550. The memory regressions, I'm afraid we're probably stuck with for a while.

> I am looking at the above alerts to see if they are related to the backout
> of screenshots or the changes made- will organize then when more data comes
> in!

There may be some noise from the backout, but they're mostly from bug 1359653.
Following improvements appeared after backout:

== Change summary for alert #6429 (as of May 06 2017 00:19 UTC) ==

Improvements:

 93%  tp5n nonmain_startup_fileio windows7-32 opt e10s     610,428.47 -> 43,022.25
 93%  tp5n nonmain_startup_fileio windows7-32 opt          608,628.83 -> 43,063.08
 31%  tsvg_static summary linux64 opt                      105.85 -> 73.37
 28%  tsvg_static summary osx-10-10 opt                    92.61 -> 66.57
 27%  tsvg_static summary windows8-64 opt                  98.57 -> 71.76
 26%  tabpaint summary windows7-32 opt                     114.04 -> 84.78
 25%  tsvg_static summary windows7-32 opt                  105.59 -> 78.78
 25%  tabpaint summary osx-10-10 opt                       116.27 -> 87.03
 20%  tabpaint summary linux64 opt                         107.80 -> 86.11
 18%  tabpaint summary windows8-64 opt                     116.24 -> 94.78
 17%  tsvg_static summary osx-10-10 opt e10s               79.20 -> 65.64
 12%  tsvg_static summary windows7-32 opt e10s             79.45 -> 70.12
 11%  tsvg_static summary windows8-64 opt e10s             68.91 -> 61.18
 10%  sessionrestore osx-10-10 opt e10s                    1,003.08 -> 904.23
 10%  tp5o summary osx-10-10 opt                           316.30 -> 285.52
 10%  sessionrestore osx-10-10 opt                         1,074.53 -> 971.33
  9%  sessionrestore_no_auto_restore osx-10-10 opt         1,099.73 -> 995.75
  9%  tpaint windows7-32 opt                               320.41 -> 290.47
  9%  tpaint osx-10-10 opt                                 402.60 -> 366.26
  9%  tpaint linux64 opt                                   295.91 -> 269.22
  9%  sessionrestore_no_auto_restore windows7-32 opt       1,004.21 -> 914.25
  9%  tpaint windows8-64 opt                               304.87 -> 277.85
  9%  tp5o summary windows8-64 opt                         364.44 -> 332.37
  9%  tart summary osx-10-10 opt                           11.64 -> 10.64
  9%  sessionrestore windows7-32 opt                       972.00 -> 888.42
  9%  sessionrestore windows8-64 opt                       895.81 -> 818.83
  8%  sessionrestore_no_auto_restore windows8-64 opt       919.25 -> 843.25
  8%  sessionrestore windows7-32 opt e10s                  958.29 -> 879.33
  8%  tp5o summary windows7-32 opt                         392.31 -> 361.10
  8%  tsvgr_opacity summary osx-10-10 opt                  452.23 -> 416.93
  8%  tart summary windows8-64 opt e10s                    6.58 -> 6.07
  8%  sessionrestore_no_auto_restore osx-10-10 opt e10s    1,013.17 -> 935.31
  7%  tps summary osx-10-10 opt                            52.80 -> 48.85
  7%  tart summary windows8-64 opt                         6.97 -> 6.46
  7%  tp5o Private Bytes windows7-32 opt e10s              158,146,888.60 -> 146,721,221.74
  7%  tp5o summary osx-10-10 opt e10s                      306.66 -> 284.83
  7%  tart summary osx-10-10 opt e10s                      11.42 -> 10.62
  7%  tp5o Main_RSS windows7-32 opt e10s                   127,861,369.29 -> 119,178,922.50
  6%  tsvgx summary windows8-64 opt                        306.67 -> 287.39
  6%  sessionrestore_no_auto_restore windows7-32 opt e10s  967.56 -> 911.00
  6%  tsvgx summary osx-10-10 opt                          486.60 -> 458.43
  6%  tp5o Private Bytes windows7-32 opt                   224,485,558.33 -> 211,727,022.96
  5%  tp5o summary windows7-32 opt e10s                    378.66 -> 357.91
  5%  tp5o Main_RSS osx-10-10 opt e10s                     358,085,490.49 -> 338,457,174.05
  5%  tp5o Main_RSS linux64 opt                            256,263,140.80 -> 243,027,865.55
  5%  tsvgr_opacity summary windows8-64 opt                397.54 -> 378.17
  5%  tp5o Main_RSS windows7-32 opt                        185,436,003.63 -> 176,656,619.25
  4%  tsvgr_opacity summary osx-10-10 opt e10s             356.50 -> 341.75
  4%  tsvgr_opacity summary windows7-32 opt                585.32 -> 561.96
  4%  ts_paint windows8-64 opt                             971.75 -> 933.17
  4%  tsvgx summary windows7-32 opt                        580.92 -> 558.45
  4%  tp5o Main_RSS osx-10-10 opt                          410,479,503.19 -> 394,877,885.12
  4%  ts_paint osx-10-10 opt e10s                          1,203.00 -> 1,158.69
  4%  ts_paint windows7-32 opt e10s                        1,088.93 -> 1,049.75
  3%  ts_paint osx-10-10 opt                               1,178.17 -> 1,139.47
  3%  damp summary windows7-32 opt e10s                    302.44 -> 293.02
  3%  tresize windows8-64 opt e10s                         11.02 -> 10.74
  2%  ts_paint windows8-64 pgo e10s                        746.68 -> 731.58

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=6429
Depends on: 1363445
After some further investigation, it looks like a lot of the startup responsiveness issues are caused by loading a huge amount of code into the Screenshots background page at startup. It would help a lot if the add-on lazily loaded its background page code when first needed (something like require.js might be a good idea), and avoided touching extension APIs before absolutely necessary.
These alerts triggered right after re-landing bug 1356243.
The sessionrestore alerts seem related to https://bugzilla.mozilla.org/show_bug.cgi?id=1356243#c17

== Change summary for alert #6922 (as of May 26 2017 19:48 UTC) ==

Regressions:

1635%  tp5n nonmain_startup_fileio windows7-32 pgo e10s     36,885.58 -> 639,886.33
1616%  tp5n nonmain_startup_fileio windows7-32 opt e10s     37,223.67 -> 638,734.92
 12%  sessionrestore windows8-64 pgo e10s                  555.92 -> 622.83
 12%  sessionrestore windows7-32 pgo e10s                  702.31 -> 786.25
 12%  sessionrestore_no_auto_restore windows8-64 pgo e10s  583.33 -> 650.42
 11%  tsvg_static summary windows8-64 opt e10s             53.43 -> 59.41
 11%  sessionrestore windows8-64 opt e10s                  680.08 -> 756.00
 11%  sessionrestore_no_auto_restore windows7-32 pgo e10s  735.35 -> 814.58
 11%  sessionrestore_no_auto_restore windows8-64 opt e10s  712.58 -> 788.75
  6%  ts_paint windows7-32 opt e10s                        994.25 -> 1,050.50
  6%  tp5o Private Bytes windows7-32 pgo e10s              141,332,004.53 -> 149,108,443.58
  5%  ts_paint windows8-64 pgo e10s                        700.00 -> 736.92
  5%  ts_paint windows8-64 opt e10s                        842.33 -> 886.58
  5%  tp5o Main_RSS windows7-32 pgo e10s                   112,940,851.46 -> 118,569,407.26
  5%  tp5o Private Bytes windows7-32 opt e10s              142,471,974.07 -> 149,064,986.05
  3%  tp5o Main_RSS windows7-32 opt e10s                   118,842,180.61 -> 122,281,956.18
  2%  tart summary windows8-64 opt e10s                    6.31 -> 6.46
  2%  tp5o_webext Main_RSS windows7-32 pgo e10s            116,577,543.62 -> 119,242,627.88

Improvements:

  8%  tsvg_static summary windows7-32 pgo e10s     59.68 -> 54.93
  5%  sessionrestore windows7-32 opt e10s          898.25 -> 855.83
  4%  sessionrestore_no_auto_restore windows7-32 opt e10s926.25 -> 884.92

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=6922
Upper link is no longer up to date.
I've grouped all alerts related to this issue on a new alert summary here:
https://treeherder.mozilla.org/perf.html#/alerts?id=6912
The version of Screenshots in https://bugzilla.mozilla.org/show_bug.cgi?id=1368146 should address these regressions.
It is good the overall regressions are much less ,and I imagine when bug 1368146 lands there will be much fewer!

here is data from AWSY, a few regressions:
== Change summary for alert #6920 (as of May 26 2017 19:48 UTC) ==

Regressions:

 11%  JS summary windows7-32-vm opt      125,150,968.92 -> 138,752,412.79
 11%  JS summary windows10-64-vm opt     174,938,295.75 -> 193,571,601.93
 11%  JS summary linux64 opt             170,207,332.50 -> 188,257,239.21
  7%  Explicit Memory summary windows10-64-vm opt 153,364,912.42 -> 164,642,924.14
  7%  Explicit Memory summary windows7-32-vm opt 118,695,485.86 -> 126,485,251.23
  6%  Explicit Memory summary linux64 opt 156,423,196.90 -> 166,308,515.19
  6%  Resident Memory summary windows7-32-vm opt 394,340,653.39 -> 417,564,718.67
  5%  Resident Memory summary windows10-64-vm opt 631,012,611.27 -> 663,032,599.84
  3%  Heap Unclassified summary windows10-64-vm opt 43,775,074.69 -> 44,947,858.21

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=6920


I assume we should wait until bug 1368146 lands before investigating these.  I would imagine memory going up, so this might just be an artifact of doing more!
That's a pretty huge memory regression.
Whiteboard: [MemShrink]
I wonder how much of that is related to lazy JS parsing being disabled in content processes (which was done as part of the work to speed up startup).
(In reply to Andrew McCreight [:mccr8] from comment #37)
> I wonder how much of that is related to lazy JS parsing being disabled in
> content processes (which was done as part of the work to speed up startup).

If that's the case, it should have shown up a couple of weeks ago rather than with this landing. I suspect this has more to do with the docshells we create for the extension's background page and the large number of scripts loaded into them. But I still wouldn't expect it to be that high.
(In reply to Joel Maher ( :jmaher) from comment #35)
> It is good the overall regressions are much less ,and I imagine when bug
> 1368146 lands there will be much fewer!
> 
> here is data from AWSY, a few regressions:
> == Change summary for alert #6920 (as of May 26 2017 19:48 UTC) ==
> 
> Regressions:
> 
>  11%  JS summary windows7-32-vm opt      125,150,968.92 -> 138,752,412.79
>  11%  JS summary windows10-64-vm opt     174,938,295.75 -> 193,571,601.93
>  11%  JS summary linux64 opt             170,207,332.50 -> 188,257,239.21
>   7%  Explicit Memory summary windows10-64-vm opt 153,364,912.42 ->
> 164,642,924.14
>   7%  Explicit Memory summary windows7-32-vm opt 118,695,485.86 ->
> 126,485,251.23
>   6%  Explicit Memory summary linux64 opt 156,423,196.90 -> 166,308,515.19
>   6%  Resident Memory summary windows7-32-vm opt 394,340,653.39 ->
> 417,564,718.67
>   5%  Resident Memory summary windows10-64-vm opt 631,012,611.27 ->
> 663,032,599.84
>   3%  Heap Unclassified summary windows10-64-vm opt 43,775,074.69 ->
> 44,947,858.21
> 
> For up to date results, see:
> https://treeherder.mozilla.org/perf.html#/alerts?id=6920
> 
> 
> I assume we should wait until bug 1368146 lands before investigating these. 
> I would imagine memory going up, so this might just be an artifact of doing
> more!

This doesn't look like the complete set of regressions, this one looks more like it:

https://treeherder.mozilla.org/perf.html#/alerts?id=6912
and we have some improvements from the screenshots update yesterday:
== Change summary for alert #6957 (as of May 30 2017 19:43 UTC) ==

Regressions:

  7%  tresize windows7-32 pgo e10s     10.70 -> 11.40
  3%  tresize windows7-32 opt e10s     12.18 -> 12.59

Improvements:

 94%  tp5n nonmain_startup_fileio windows7-32 pgo e10s     637,885.38 -> 35,862.83
 94%  tp5n nonmain_startup_fileio windows7-32 opt e10s     637,370.92 -> 36,884.25
 11%  ts_paint_webext linux64 opt e10s                     1,625.25 -> 1,438.58
  9%  sessionrestore windows8-64 opt e10s                  745.83 -> 679.33
  9%  sessionrestore_no_auto_restore windows8-64 opt e10s  775.00 -> 708.25
  7%  sessionrestore_no_auto_restore osx-10-10 opt e10s    1,135.50 -> 1,051.58
  7%  sessionrestore osx-10-10 opt e10s                    1,103.71 -> 1,024.25
  7%  sessionrestore linux64 opt e10s                      890.83 -> 831.75
  5%  sessionrestore_no_auto_restore linux64 opt e10s      917.58 -> 870.17
  4%  ts_paint linux64 opt e10s                            1,357.25 -> 1,297.25
  4%  ts_paint windows8-64 opt e10s                        871.17 -> 833.92

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=6957



here is what remains from 6912 (I removed the ones that improved above):
== Change summary for alert #6912 (as of May 26 2017 19:48 UTC) ==

Regressions:

 28%  tp5o responsiveness linux64 opt e10s                 5.16 -> 6.63
 13%  tp5o_webext responsiveness linux64 opt e10s          5.94 -> 6.70
 11%  tsvg_static summary windows8-64 opt e10s             53.43 -> 59.41
 11%  tsvg_static summary windows8-64 pgo e10s             50.75 -> 56.39
  6%  tp5o Main_RSS linux64 opt e10s                       176,881,345.14 -> 186,852,041.43
  6%  tp5o Private Bytes windows7-32 pgo e10s              141,332,004.53 -> 149,108,443.58
  5%  tp5o Main_RSS windows7-32 pgo e10s                   112,940,851.46 -> 118,569,407.26
  5%  tp5o Private Bytes windows7-32 opt e10s              142,471,974.07 -> 149,064,986.05
  3%  tp5o_webext Private Bytes windows7-32 opt e10s       146,083,241.36 -> 150,860,199.26
  3%  tp5o Main_RSS windows7-32 opt e10s                   118,842,180.61 -> 122,281,956.18
  3%  tp5o_webext Main_RSS linux64 opt e10s                183,161,911.37 -> 188,384,042.80
  2%  tart summary windows8-64 opt e10s                    6.31 -> 6.46
  2%  tart summary linux64 opt e10s                        6.54 -> 6.70
  2%  tp5o_webext Main_RSS windows7-32 pgo e10s            116,577,543.62 -> 119,242,627.88
  2%  tp5o Private Bytes linux64 opt e10s                  1,164,649,729.85 -> 1,188,128,400.30
+
  7%  tresize windows7-32 pgo e10s     10.70 -> 11.40
  3%  tresize windows7-32 opt e10s     12.18 -> 12.59


if you ignore memory, this is what remains:
 28%  tp5o responsiveness linux64 opt e10s                 5.16 -> 6.63
 13%  tp5o_webext responsiveness linux64 opt e10s          5.94 -> 6.70
 11%  tsvg_static summary windows8-64 opt e10s             53.43 -> 59.41
 11%  tsvg_static summary windows8-64 pgo e10s             50.75 -> 56.39
  2%  tart summary windows8-64 opt e10s                    6.31 -> 6.46
  2%  tart summary linux64 opt e10s                        6.54 -> 6.70
  7%  tresize windows7-32 pgo e10s     10.70 -> 11.40
  3%  tresize windows7-32 opt e10s     12.18 -> 12.59


it looks like focusing on svg_static and looking at responsiveness would be the next big wins, likewise any memory work.
The improvements in the startup numbers were caused by delaying the loading of screenshots until after sessionrestore finishes. The tsvg and responsiveness regressions are almost certainly caused by loading and initializing the extension after that.

Bug 1322235 should help with that, but the biggest culprit at this point is loading the extension's background scripts, and generating bindings for them. Moving to OOP will hide most of that overhead, because it would happen in a child process, but it still needs to be dealt with.

The extension really needs to start with a skeleton background page that only loads the scripts and touches the APIs that it needs immediately at startup. Unfortunately, I don't think we can avoid it touching at least browserAction.onClicked, runtime.onMessage, i18n.getMessage, and contextMenus.create, but if we can get away with only touching those APIs from a single background script, until some Screenshots functionality is activated, I think that's what we should do.
I tried removing all the background scripts as a test.  Once we put deferred startup in, the background scripts didn't seem to have any effect on performance: https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=0ba0c9e23b8652874a640e042ca2a87a47d48c27&newProject=try&newRevision=55da0f61470cffaefded54be2f44dbf0462caec3&framework=1&showOnlyImportant=0

Maybe something has landed that would change this, I'm not sure.  This commit was my simulation of lazy loading the background page: https://hg.mozilla.org/try/rev/6d0910ce3662
[Tracking Requested - why for this release]: Performance regression
Severity: normal → major
status-firefox53: --- → unaffected
status-firefox54: --- → unaffected
status-firefox55: --- → affected
status-firefox-esr45: --- → unaffected
status-firefox-esr52: --- → unaffected
tracking-firefox55: --- → ?
Keywords: topperf
OS: Unspecified → All
Hardware: Unspecified → All
Version: 53 Branch → 55 Branch
(In reply to Ian Bicking (:ianb) from comment #42)
> I tried removing all the background scripts as a test.  Once we put deferred
> startup in, the background scripts didn't seem to have any effect on
> performance:
> https://treeherder.mozilla.org/perf.html#/
> compare?originalProject=try&originalRevision=0ba0c9e23b8652874a640e042ca2a87a
> 47d48c27&newProject=try&newRevision=55da0f61470cffaefded54be2f44dbf0462caec3&
> framework=1&showOnlyImportant=0
> 
> Maybe something has landed that would change this, I'm not sure.  This
> commit was my simulation of lazy loading the background page:
> https://hg.mozilla.org/try/rev/6d0910ce3662

Hmm, that's weird.  Have you tried profiling a run with and without the screenshot pref turned on to see what changes when we have the pref enabled to get a sense of where the additional time is being spent?  That may help shed some more light into this, especially given the above.
I did a try run a few days ago with most Screenshots background scripts removed, and it made a clear difference in responsiveness numbers:

https://treeherder.mozilla.org/perf.html#/compare?originalProject=mozilla-inbound&originalRevision=118e1c11be5c&newProject=try&newRevision=55aece2bfb861c8597682bc7d8b877585c40e7d3&framework=1&filter=respon&showOnlyImportant=0
https://treeherder.mozilla.org/perf.html#/compare?originalProject=mozilla-inbound&originalRevision=0ea05f32de96&newProject=try&newRevision=55aece2bfb861c8597682bc7d8b877585c40e7d3&framework=1&filter=resp&showOnlyImportant=0

Though it doesn't make up for the entire regression.

Unfortunately, those numbers are really hard to interpret, since they could mean 1ms of event loop lag pretty consistently (not very bad), or a few instances where we have 10ms of event loop lag (bad), or a couple of instances where we have 500ms of event loop lag (Very Bad).

But regardless of the numbers, we need to avoid eagerly loading those scripts before they're necessary. The last time I profiled this on a fast machine, we were spending something like 40-50ms loading those background scripts, in fairly large chunks, which means we're probably creating a lot of jank just after startup (which is generally a time we want to particularly avoid creating jank).
I've reopened https://github.com/mozilla-services/screenshots/issues/2843 to discuss finishing the implementation of lazy-loading the Screenshots background page
Tracking this perf issue for 55.
tracking-firefox55: ? → +
The v10 release of Screenshots (Bug 1372310) will add lazy loading of the bulk of the background page scripts
Depends on: 1372310
Whiteboard: [MemShrink] → [MemShrink:P1]
Hi Ian, Eric, Mahe: When will v10 of screenshots be uplifts to Beta55? How soon after landing this can we have an official talos run that compares the memusage results to determine how substantial the improvements are and whether screenshots is ok to go in Beta55?
Flags: needinfo?(mpotharaju)
Flags: needinfo?(ianb)
Flags: needinfo?(erahm)
Hey Ritu, We have a new version v10.3 ready, will request an uplift next week after QA. Will share the necessary talos runs and test results in a status email.
Flags: needinfo?(mpotharaju)
Flags: needinfo?(ianb) → needinfo?(jhirsch)
Flags: needinfo?(jhirsch)
What's the status on the uplift? Thanks!
Flags: needinfo?(erahm) → needinfo?(mpotharaju)
Flag set and uplift requested for Beta 4. Will update the bug once it makes in Beta 4 on 6/22.
Mike, Screenshots is in 56 now. Pref on to 1% users today.
Flags: needinfo?(mpotharaju)
Sorry. Correction, its 55.
Jeff, what's the call here for 55?
Flags: needinfo?(jgriffiths)
Summary: 2.11 - 48.22% most all tests (linux64, osx-10-10, windows7-32, windows8-64) regression on push 8090b323f5d1 (Wed May 3 2017) → (Firefox Screenshots) 2.11 - 48.22% most all tests (linux64, osx-10-10, windows7-32, windows8-64) regression on push 8090b323f5d1 (Wed May 3 2017)
(In reply to Mike Taylor [:miketaylr] (55 Regression Engineering Owner) from comment #55)
> Jeff, what's the call here for 55?

+1 from me as long as relman is happy.
Flags: needinfo?(jgriffiths)
It seems like all signals point to us being happy to ship this, given that I'm gonna remove the regression keyword so we don't have to keep looking at this in regression triage.
Keywords: regression
(In reply to Mike Taylor [:miketaylr] (55 Regression Engineering Owner) from comment #57)
> It seems like all signals point to us being happy to ship this, given that
> I'm gonna remove the regression keyword so we don't have to keep looking at
> this in regression triage.

You can just simply update the tracking flag of status-firefox55 to fixed so that it will be removed from 55 regression tracking radar.
That makes sense, let's do that (the perf regressions has been fixed... or is at least acceptable enough to ship with).
status-firefox55: affected → fixed
Keywords: regression
Looks like all the blockers are fixed.
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.