Closed Bug 1292721 Opened 8 years ago Closed 8 years ago

[e10s] Increase e10s activation on the release channel to 10% of eligible users

Categories

(Firefox :: General, defect)

48 Branch
defect
Not set
normal

Tracking

()

RESOLVED FIXED
Tracking Status
firefox48 + fixed

People

(Reporter: Felipe, Assigned: Felipe)

References

Details

Attachments

(4 files)

+++ This bug was initially created as a clone of Bug #1284958 +++

If everything is looking good, the plan is to do this via a GoFaster update of the e10srollout add-on in or around Aug 12th. Until there, let's get the add-on prepared, signed, tested and ready to go.
Felipe, Hello already has a planned Go Faster update on 15th August. Since we don't normally push out release on a Friday, do you want to sync up & release on the 15th at the same time as Hello?

You'll also need to notify release-drivers of the intention as well:

https://wiki.mozilla.org/Firefox/Go_Faster/Process#Process_to_Push_updates_to_release_channel
Flags: needinfo?(felipc)
Hi Mark, thanks for letting me know. I'll bring this up with the e10s team to see if they are ok with it
Flags: needinfo?(felipc)
Plan of record is to actually do it sooner (hopefully by tomorrow, Aug 9th)
Depends on: 1293372
Comment on attachment 8778990 [details]
[mq]: Bug 1292721 - Increase e10s activation on the release channel to 10% of eligible users.

https://reviewboard.mozilla.org/r/70058/#review67342

\o/
Attachment #8778990 - Flags: review?(mconley) → review+
Depends on: 1293398
Depends on: 1293418
Michael, can you handle this rollout for me, I'm meant to be on PTO this week.

Please re-use the "SystemAddons-ff48-rel1" and downgrade loop to the 1.4.3 that Felipe is uploading & downgrade e10srollout. I'll create a rel2 next week for the loop release.

Felipe, whilst I realise the change to the add-on is normally simple, generally we have QA sign-off on the update before we publish it. Have you got QA set up to do that?
(In reply to Mark Banner (:standard8) (afk 8 - 12 August) from comment #7)
> Michael, can you handle this rollout for me, I'm meant to be on PTO this
> week.
> 

Hey Mark (sorry to bug you on PTO) - was there supposed to be a needinfo on this one? Which Michael are you referring to?
Flags: needinfo?(standard8)
Comment on attachment 8778990 [details]
[mq]: Bug 1292721 - Increase e10s activation on the release channel to 10% of eligible users.

Approval Request Comment:

Sylvestre, whenever there's an update, Firefox reverts back to the bundled system add-ons (vs. the ones updated via GoFaster). This means that even after we release the 10% update to 48.0 users, we need to ship this patch built-in with the 48.0.1 dot release in order to continue with 10% (otherwise it will revert to 1%)
Attachment #8778990 - Flags: approval-mozilla-release?
(I think he meant Michael Kelly)
Flags: needinfo?(standard8) → needinfo?(mkelly)
Alright, I tested locally a system add-on set update with the newly signed xpi, the right pocket xpi and also a newer version of loop. Once we get bug 1293398 and bug 1293418, we can test the update on a staging location.
I don't think QA is necessary for this but if people want it I can ask QA to do some sanity checks across platforms tomorrow.
Andrei, could you help with this system addon verification?
Flags: needinfo?(andrei.vaida)
Comment on attachment 8778990 [details]
[mq]: Bug 1292721 - Increase e10s activation on the release channel to 10% of eligible users.

Let's take it to the release branch
Attachment #8778990 - Flags: approval-mozilla-release? → approval-mozilla-release+
(In reply to Sylvestre Ledru [:sylvestre] from comment #12)
> Andrei, could you help with this system addon verification?

Sure. Felipe, what should we focus on here? We'll definitely need the signed *.xpi and the staging location before hand, but I'm particularly concerned about how we're going to confirm the increase to 10% (if that's actually expected of QA).
Flags: needinfo?(andrei.vaida) → needinfo?(felipc)
Hi Andrei, thanks for the offer. Here's an step by step on all that should be tested:

1. Using a clean profile from Firefox release 48.0

2. Go to about:support and verify:

 - Multiprocess Windows: 0/1 (Disabled) 

 - Installed extensions:
   - Firefox Hello Beta, version 1.4.3
   - Pocket, version 1.0.4
   - Multi-process staged rollout, 1.0

3. Go to about:config and change the following:
  - devtools.chrome.enabled, set to true
  - e10s.rollout.cohortSample, set to "0.09" (no quotes)
  - extensions.systemAddon.update.url set to "http://people.mozilla.org/~fgomes/e10s-update/update-ff48-1.1.xml" (no quotes)

4. Force an add-on update check, by running the following in Tools > Web Developer > Browser Console:

    Components.utils.import("resource://gre/modules/AddonManager.jsm");
    AddonManagerPrivate.backgroundUpdateCheck();

5. Wait a minute in order for the update to download, then restart

6. After a restart, check in about:support

  - Multiprocess Windows: 1/1 (Enabled)

  - Installed extensions:
   - Firefox Hello Beta, version 1.4.3   (no change)
   - Pocket, version 1.0.4   (no change)
   - Multi-process staged rollout, 1.1    (* changed *)

7. And then do some basic sanity check that Hello and Pocket works
Flags: needinfo?(felipc)
Attached file update.xml
Ben, all the xpis are signed and uploaded to their correct locations, so I manually created the update.xml to make this easier (with the right file sizes and checksums).

I tested an update with this file locally, and now Andrei will test it remotely that I uploaded to my people.mozilla.org. But if you can get it in any staging form from Balrog we could test it from there instead. Not sure how important that is
Attachment #8779392 - Flags: feedback?(bhearsum)
I've been having trouble connecting to Balrog since the AWS transition, so until that gets resolved (bhearsum is filing a bug to fix it at this very moment) he'll be on point for performing this rollout.
Flags: needinfo?(mkelly)
(In reply to :Felipe Gomes (needinfo me!) from comment #16)
> Created attachment 8779392 [details]
> update.xml
> 
> Ben, all the xpis are signed and uploaded to their correct locations, so I
> manually created the update.xml to make this easier (with the right file
> sizes and checksums).
> 
> I tested an update with this file locally, and now Andrei will test it
> remotely that I uploaded to my people.mozilla.org. But if you can get it in
> any staging form from Balrog we could test it from there instead. Not sure
> how important that is

Okay, I've done this on the "release-sysaddon" update channel. Any System Addon update requests done via that channel from Firefox 48.0 should receive:
- Loop 1.4.3 (same as built in)
- Pocket 1.0.4 (same as built in)
- E10S Rollout 1.1
Thanks Ben. Andrei, you can change the value for the extensions.systemAddon.update.url from comment 15, and use this one instead:

https://aus5.mozilla.org/update/3/SystemAddons/%VERSION%/%BUILD_ID%/%BUILD_TARGET%/%LOCALE%/release-sysaddon/%OS_VERSION%/%DISTRIBUTION%/%DISTRIBUTION_VERSION%/update.xml

(note: it's similar to the default pref, but it's indeed a bit different)
Flags: needinfo?(andrei.vaida)
Pushed the increase to 10% to mozilla-release so that it's built-in for 48.0.1:

https://hg.mozilla.org/releases/mozilla-release/rev/086d9208b431
Attachment #8779392 - Flags: feedback?(bhearsum)
We can confirm that the Multi-process staged rollout sys add-on was
successfully updated to v1.1 via release-sysaddon channel. Two
potential issues were found while testing this:

  1. the logs displayed after forcing the add-on update check
  (Comment 15, Step 4) are not reflecting the fact that an update
  to 1.1 is actually found, (e.g.): 

     	1470816580665 addons.update-checker WARN Update manifest for
     	{972ce4c6-7e08-4474-a285-3208198ce6fd} did not contain an
     	updates property

	1470816580884 addons.update-checker WARN Update manifest for
	e10srollout@mozilla.org did not contain an updates property

	1470816581324 addons.update-checker WARN Update manifest for
	firefox@getpocket.com did not contain an updates property
  
  2. after restarting the browser (Comment 15, Step 6) the language
  used throughout the UI of Hello is Portuguese
  	- just to be clear, at first startup, the language of Hello was
  	English

Please note that everything else works properly and no functional
issues were found while spot-checking Hello 1.4.3 and Pocket 1.0.4.
Flags: needinfo?(andrei.vaida) → needinfo?(felipc)
(In reply to Andrei Vaida, QA [:avaida] – please ni? me from comment #21)
>   2. after restarting the browser (Comment 15, Step 6) the language
>   used throughout the UI of Hello is Portuguese
>   	- just to be clear, at first startup, the language of Hello was
>   	English

Hmm that's not good.. I thought that all languages were bundled in the built-in package that I got from release 48. I guess to properly build 1.4.3 it would have to be from the github repo? But we would need an answer from Standard8 or someone else who knows what's the right thing to do
Flags: needinfo?(standard8)
Flags: needinfo?(ianb)
Flags: needinfo?(felipc)
rhelmer may have some insight about using the XPIs built during the Firefox build process, he's been advocating that for a while.
Flags: needinfo?(rhelmer)
The pocket-1.0.4 signed xpi that was already uploaded (I imagine in preparation for next week's GoFaster release) also only contains en-US. So I suspect the same prob would happen with it but it would go unnoticed due to being en-US.
(In reply to Michael Kelly [:mkelly,:Osmose] from comment #23)
> rhelmer may have some insight about using the XPIs built during the Firefox
> build process, he's been advocating that for a while.

There are a few bugs open for this (bug 1277920 and bug 1281578), but I know Hello's current process is a little different so we probably need to wait for ianb/standard8 to answer comment 22.
Following instructions for AMO release except failing on `upload_xpi` for `JPM_API_KEY and JPM_API_SECRET should be defined`:
https://github.com/mozilla/loop/blob/master/docs/Releases.md#shipping-to-amo

$ git clone git@github.com:mozilla/loop.git
$ cd loop
$ git checkout v1.4.3
$ make distclean
$ make dist
$ ls dist/loop\@mozilla.org.xpi

Attached is the xpi from the last line.
(In reply to Ed Lee :Mardak from comment #26)
> This xpi only has one packaged locale, pt-BR:
> 
> https://ftp.mozilla.org/pub/system-addons/hello/loop@mozilla.org-ff48-1.4.3-
> signed.xpi

yeah, this is how the issue was fortunately found by QA. The other problem now is that http://ftp.mozilla.org/pub/system-addons/pocket/firefox@getpocket.com-ff48-1.0.4-signed.xpi only has en-US
(In reply to Ed Lee :Mardak from comment #27)
> Created attachment 8779778 [details]
> v1.4.3 loop@mozilla.org.xpi
Compared to bug 1291753's attachment 8777391 [details], the only differences are indeed just the changes between the tagged versions: https://github.com/mozilla/loop/compare/v1.4.3...v1.4.4

Should I file a bug for signing or we're waiting on the pocket not-just-en-US xpi?
(In reply to Ed Lee :Mardak from comment #29)
> (In reply to Ed Lee :Mardak from comment #27)
> > Created attachment 8779778 [details]
> > v1.4.3 loop@mozilla.org.xpi
> Compared to bug 1291753's attachment 8777391 [details], the only differences
> are indeed just the changes between the tagged versions:
> https://github.com/mozilla/loop/compare/v1.4.3...v1.4.4
> 
> Should I file a bug for signing or we're waiting on the pocket
> not-just-en-US xpi?

Thanks, and yes please. In the same bug, ask for it to be uploaded to ftp, replacing the current https://ftp.mozilla.org/pub/system-addons/hello/loop@mozilla.org-ff48-1.4.3-signed.xpi

Since I'm not involved with the Hello code I didn't want to personally make a call, so I asked Ian Bicking and he said the xpi looks correct.
Oh yes, I forgot that shipping FFs only include the one translation for Hello. Using the packaged one (or the 1.4.3 one from AMO is the right thing to do).

For pocket, I don't know what to do. I thought the xpi didn't include the locales, and they were shipped as part of Firefox. It appears that isn't the case. So it looks like the best that can be done is to manually create an xpi from the locales in m-r.
Flags: needinfo?(standard8)
Interesting, is that signature also valid for system add-ons? Jorge do you know?
Flags: needinfo?(jorge)
Flags: needinfo?(ianb)
(In reply to Andrei Vaida, QA [:avaida] – please ni? me from comment #21)
> We can confirm that the Multi-process staged rollout sys add-on was
> successfully updated to v1.1 via release-sysaddon channel. Two
> potential issues were found while testing this:
> 
>   1. the logs displayed after forcing the add-on update check
>   (Comment 15, Step 4) are not reflecting the fact that an update
>   to 1.1 is actually found, (e.g.): 
> 
>      	1470816580665 addons.update-checker WARN Update manifest for
>      	{972ce4c6-7e08-4474-a285-3208198ce6fd} did not contain an
>      	updates property
> 
> 	1470816580884 addons.update-checker WARN Update manifest for
> 	e10srollout@mozilla.org did not contain an updates property
> 
> 	1470816581324 addons.update-checker WARN Update manifest for
> 	firefox@getpocket.com did not contain an updates property

These logs are different, there are two checks happening here:

1) AMO is checked for updates (and compatibility info), or a custom URL if the updates property is specified

2) System Add-ons are checked against AUS (aka Balrog)
Flags: needinfo?(rhelmer)
Manually packaged all fully translated pocket locales. This is untested yet. I'll file a bug for signing and uploading so that we can test.
Flags: needinfo?(jorge)
Depends on: 1294214
Andrei, we're now ready to repeat the same tests from comment 15, but please use the following URL for the extensions.systemAddon.update.url pref:

http://people.mozilla.org/~fgomes/e10s-update/update.xml


The focus here is to do a sanity check on Hello and Pocket and make sure they continue to work properly, specially in a different locale.

Please test the following:
 - with en-US build, it should work and remain in English
 - with de build, it should work and remain in German
 - with vi build, it should work and be presented in English (since there are no strings translated for Hello and Pocket in Vietnamese)
Flags: needinfo?(andrei.vaida)
We finished running our tests again, using the three locales and the
new update URL (Comment 36) and _almost_ everything is working
properly:

    - the strings used for Hello and Pocket are now correctly used
    based on locale
    - the core functionality of Hello and Pocket is intact
    - Multi-process staged rollout has been successfully updated to
    v1.1

Please note that, compared to the last time we ran these tests:

    - we're still seeing those logs after forcing the add-on update
    check (see Comment 34) along with the following new log:

	1470905616075 addons.xpi ERROR Attempted to load bootstrap scope
	from missing directory

    - we also noticed that before updating the e10s rollout sys
    add-on, Hello was displayed as "Firefox Hello, version 1.4.3" and
    after the update it's displayed as "Firefox Hello Beta, version
    1.4.3" (notice the "Beta" additional word added to its name)
Flags: needinfo?(andrei.vaida) → needinfo?(felipc)
As Hello is going to die, I don't think that as a big issue...
For the log, I don't know.
Thanks Andrei. The name change is not an issue. For the log, I believe it's also unrelated, but could you toggle the pref "extensions.logging.enabled" to true? It will give a lot more output, and right before that message there should be another one saying: "Loading bootstrap scope from ..." which will give us more info.

(I'll look into it myself too after a meeting)

I imagine this message is just due to the built-in system add-on being replaced by the downloaded one.
Flags: needinfo?(felipc)
Flags: needinfo?(andrei.vaida)
Ok, I looked into it and that message is coming from the hotfix code, so unrelated to system addons. 

1470933062904	addons.xpi	DEBUG	Loading bootstrap scope from /Users/felipe/moz/release/ff-opt/tmp/scratch_user/extensions/firefox-hotfix@mozilla.org.xpi
1470933062905	addons.xpi	ERROR	Attempted to load bootstrap scope from missing directory /Users/felipe/moz/release/ff-opt/tmp/scratch_user/extensions/firefox-hotfix@mozilla.org.xpi

There's a bunch of pre-existing warnings related to hotfix that I'll file a bug about it. But it is unrelated to this update.
This was just pushed live. Thanks to everyone involved here.

Now that the system add-ons from 48 are all properly packaged as standalone xpis, the next e10srollout updates during this cycle should be much simpler to do.
Status: ASSIGNED → RESOLVED
Closed: 8 years ago
Flags: needinfo?(andrei.vaida)
Resolution: --- → FIXED
Blocks: 1299247
You need to log in before you can comment on or make changes to this bug.