Ionuț Goldan [:igoldan] Reporter
	Description • 6 years ago

We have detected an awsy regression from push:

https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=9322f0ce835f554a0108f59dba244f32ebd8d492&tochange=90ad57245468c398be0be3fcdb2cdec1b9333232

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

 23%  Images windows10-64 opt stylo     7,105,802.51 -> 8,765,477.90
 22%  Images windows10-64 pgo stylo     7,462,464.75 -> 9,090,809.38
  4%  Heap Unclassified windows10-64 pgo stylo41,276,303.39 -> 43,113,613.28
  4%  Heap Unclassified windows10-64 opt stylo41,327,290.70 -> 43,066,678.10
  4%  Resident Memory windows10-64 opt stylo473,359,269.95 -> 490,130,107.88
  4%  Explicit Memory windows10-64 opt stylo311,803,364.88 -> 322,762,749.88
  3%  Resident Memory windows10-64 pgo stylo464,851,298.54 -> 480,887,147.40
  3%  Explicit Memory windows10-64 pgo stylo312,704,988.06 -> 323,460,034.89


You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=11997

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://wiki.mozilla.org/AWSY/Tests

	Ionuț Goldan [:igoldan] Reporter
	Comment 1 • 6 years ago

Other platforms are affected as well:

Regressions:

 27%  Images windows7-32 opt stylo     5,693,513.40 -> 7,225,337.57
 26%  Images linux64-stylo-sequential opt stylo-sequential6,140,888.39 -> 7,729,311.24
 24%  Images osx-10-10 opt stylo       6,498,574.26 -> 8,042,625.19
  8%  Explicit Memory osx-10-10 opt stylo322,331,602.85 -> 348,534,520.90
  6%  Heap Unclassified windows7-32 opt stylo34,326,129.16 -> 36,303,555.68
  6%  Heap Unclassified windows7-32 pgo stylo34,171,244.88 -> 36,113,536.48
  4%  Explicit Memory windows7-32 opt stylo252,345,702.18 -> 262,951,473.47
  4%  Explicit Memory windows7-32 pgo stylo252,441,930.67 -> 262,428,847.34
  3%  Explicit Memory linux64-stylo-sequential opt stylo-sequential310,074,352.13 -> 319,732,109.16
  2%  Explicit Memory linux64 opt stylo313,311,254.14 -> 320,752,266.83

	Ionuț Goldan [:igoldan] Reporter
	Comment 2 • 6 years ago

:Mardak It starting to look like bug 1383599 is responsible for these serious AWSY perf regressions. Please look over this issue.

I've retriggered the perf tests, but the results just started to show up.

	Ed Lee :Mardak Assignee
	Comment 3 • 6 years ago

It looks like the cause for the regression is that AWSY is running with browser.newtabpage.enabled = false. Where before bug 1383599, it was running as if browser.newtabpage.enabled = true (because the pref wasn't being checked).

erahm -> jmaher, does the test need to run with enabled = true, and is this where it's being set:

https://searchfox.org/mozilla-central/rev/588d8120aa11738657da93e09a03378bcd1ba8ec/testing/marionette/components/marionette.js#89-94

And should it be flipped to true (just remove it as the default is true) as that's what it's been running with?

	Joel Maher ( :jmaher ) (UTC -8)
	Comment 4 • 6 years ago

I am not sure here and :erahm is on pto for another week.  if setting this =True fixes this regression we should do that and then leave a pending issue for :erahm.

	Ed Lee :Mardak Assignee
	Comment 5 • 6 years ago

Try run https://treeherder.mozilla.org/perf.html#/compare?originalProject=mozilla-central&originalRevision=d55b703721d2&newProject=try&newRevision=88fc24d19259&framework=4&filter=osx

Shows -25.97% Images osx (for 1 run before and after)

	Joel Maher ( :jmaher ) (UTC -8)
	Comment 7 • 6 years ago
mozreview-review

Comment on attachment 8957665 [details]
Bug 1444404 - Use default (true) value for browser.newtabpage.enabled for testing.

https://reviewboard.mozilla.org/r/226586/#review232440

I am not sure what other tests will be affected by this, I suspect quite a few.  Have you tested this on all unittests (try -bo -p all -u all)?

	Pulsebot
	Comment 9 • 6 years ago

Pushed by edilee@gmail.com:
https://hg.mozilla.org/integration/autoland/rev/9a200b344efe
Use default (true) value for browser.newtabpage.enabled for testing. r=jmaher

	Noemi Erli[:noemi_erli]
	Comment 10 • 6 years ago
bugherder

https://hg.mozilla.org/mozilla-central/rev/9a200b344efe

	Andreas Tolfsen ❲:ato❳
	Comment 11 • 6 years ago

This patch made a change to files for automation tooling that targets
out-of-tree release channels without the review from a
geckodriver/Marionette peer.  I’m adding some notes to the relevant
files to prevent this from happening in the future as part of bug
1401129.

As I understand it browser.newtabpage.enable = false was broken at
least since Firefox 56?  Do we know about Firefox 55?

My assumption is that it we don’t need to add this pref back for
Firefox 55 compatibility, but I’d like to check to be sure.

	Ed Lee :Mardak Assignee
	Comment 12 • 6 years ago

Just checked 55.0.3, and browser.newtabpage.enabled does set the page to be mostly blank for the default old Tiles about:newtab, and activity stream isn't packaged, so browser.newtabpage.activity-stream.enabled does nothing.

And in 56.0.2, browser.newtabpage.enabled also shows a mostly blank page for old Tiles about:newtab, and now activity stream is packaged

For 57.0.4, activity stream is the default about:newtab, and browser.newtabpage.enabled doesn't do anything there.

So the pref was not entirely functional since 57 (although one could trigger the blank by disabling both activity stream and newtabpage).

	Andreas Tolfsen ❲:ato❳
	Comment 13 • 6 years ago

Thanks for checking, mardak!  I don’t think we should have to worry
about this for the geckodriver 0.20.0 release.

	Ionuț Goldan [:igoldan] Reporter
	Comment 14 • 6 years ago

I confirm the push from comment 9 resolved this bug's issue:

== Change summary for alert #12052 (as of Fri, 09 Mar 2018 21:17:16 GMT) ==

Improvements:

 24%  Images windows7-32 pgo stylo     7,615,412.06 -> 5,771,764.67
 21%  Images linux64-stylo-sequential opt stylo-sequential7,713,135.16 -> 6,126,343.62
 20%  Images linux64 opt stylo         7,760,082.68 -> 6,170,993.01
 20%  Images windows10-64 opt stylo    9,022,122.53 -> 7,213,663.75
 19%  Images windows10-64 pgo stylo    9,182,175.45 -> 7,436,880.46
 19%  Images windows7-32 opt stylo     7,119,213.66 -> 5,766,443.51
 17%  Images osx-10-10 opt stylo       7,792,453.23 -> 6,439,950.40
  5%  Heap Unclassified windows7-32 pgo stylo35,995,663.55 -> 34,084,058.05
  5%  Heap Unclassified windows7-32 opt stylo36,214,134.73 -> 34,314,121.08
  4%  Heap Unclassified windows10-64 pgo stylo43,076,576.72 -> 41,142,084.77
  4%  Explicit Memory windows7-32 pgo stylo263,656,416.52 -> 252,222,456.27
  4%  Heap Unclassified windows10-64 opt stylo42,931,024.60 -> 41,118,374.97
  4%  Explicit Memory windows10-64 pgo stylo325,134,727.97 -> 312,914,421.07
  4%  Explicit Memory windows7-32 opt stylo262,229,848.37 -> 252,390,323.57
  4%  Explicit Memory windows10-64 opt stylo323,345,299.33 -> 311,483,309.89
  3%  Explicit Memory osx-10-10 opt stylo352,670,241.26 -> 340,546,597.13
  3%  Explicit Memory linux64-stylo-sequential opt stylo-sequential320,146,774.54 -> 309,550,103.27
  3%  Resident Memory windows10-64 pgo stylo481,836,990.81 -> 466,890,741.34
  3%  Resident Memory windows10-64 opt stylo490,748,101.23 -> 475,774,845.05
  3%  Heap Unclassified osx-10-10 opt stylo74,111,632.79 -> 71,902,783.67
  3%  Explicit Memory linux64 opt stylo321,628,888.69 -> 313,598,628.41

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=12052