Closed Bug 1339870 Opened 7 years ago Closed 7 years ago

Localised stub installer for Nightly installs en-US Firefox

Categories

(Release Engineering :: General, defect, P1)

Product:
Release Engineering
Component:
General
Type:
defect

Tracking

(firefox51 unaffected, firefox52 unaffected, firefox53+ unaffected, firefox54 unaffected, firefox55+ fixed)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox51 --- unaffected
firefox52 --- unaffected
firefox53 + unaffected
firefox54 --- unaffected
firefox55 + fixed

People

(Reporter: lizzard, Assigned: jlorenzo)

References

Details

Attachments

(1 file)

product delivery tools PR
66 bytes, text/x-github-pull-request
oremj
: review+
Details | Review 
Filing this for Camelia who reports that on testing the win 64 stub installer on Aurora 53, all locales are updating to English. (Using builds from https://archive.mozilla.org/pub/firefox/nightly/latest-mozilla-aurora-l10n/ )

May be a problem in the bouncer rules, or something else.
Matt adds,
 
Reproducible from here. Looks like something's off in the Bouncer rules. The installer requests the correct URL, for instance:
http://download.mozilla.org/?os=win64&lang=es-ES&product=firefox-aurora-latest

but the response is a redirect to the en-US build at:
http://download.cdn.mozilla.net/pub/firefox/nightly/latest-mozilla-aurora/firefox-53.0a2.en-US.win64.installer.exe
So, for aurora we have to separate en-US and l10n bouncer products, because they live in different directories. 

The actual link should be http://download.mozilla.org/?os=win64&lang=es-ES&product=firefox-aurora-latest-l10n

I also tried to install Aurora using a localized stub installer. The setup interface was localized, but the final installed version was en-US, probably because of the bouncer link.
(In reply to Rail Aliiev [:rail] ⌚️ET from comment #2)
> So, for aurora we have to separate en-US and l10n bouncer products, because
> they live in different directories. 
> 
> The actual link should be
> http://download.mozilla.org/?os=win64&lang=es-ES&product=firefox-aurora-
> latest-l10n
> 
> I also tried to install Aurora using a localized stub installer. The setup
> interface was localized, but the final installed version was en-US, probably
> because of the bouncer link.

Rail, when you say "bouncer link", you are referring to the URL hardcoded in the stub installer code and not the CDN URL returned from the bouncer server, right? So this is a problem Matt will need to fix in the stub installer?

To catch URL problems like this, could bouncer return an error if a request specifies a non-"en-US" `lang` with a `product` that doesn't end in "-l10n" (and vice versa)?
Blocks: support-win64
status-firefox51: --- → unaffected
status-firefox52: --- → unaffected
status-firefox54: --- → ?
Flags: needinfo?(rail)
(In reply to Chris Peterson [:cpeterson] from comment #3)
> Rail, when you say "bouncer link", you are referring to the URL hardcoded in
> the stub installer code and not the CDN URL returned from the bouncer
> server, right?

Correct, these ones: https://dxr.mozilla.org/mozilla-aurora/source/browser/branding/aurora/branding.nsi#17-18

BTW, this issue is probably the same for Nightly. :(

> So this is a problem Matt will need to fix in the stub installer?

Ugh... It sounds very dirty to have special logic for localized builds... I wish we published the l10n nightly builds in the same directory with en-US... Changing the current location will definitely break some automation. Maybe we should copy en-US to the latest-l10n directory and use that location in bouncer.

Summoning Nick, he may have brighter ideas. :)

> To catch URL problems like this, could bouncer return an error if a request
> specifies a non-"en-US" `lang` with a `product` that doesn't end in "-l10n"
> (and vice versa)?

To be honest, I would prefer not to have special logic for this in bouncer. :(
Flags: needinfo?(rail) → needinfo?(nthomas)
(In reply to Rail Aliiev [:rail] ⌚️ET from comment #4)
> BTW, this issue is probably the same for Nightly. :(

And it doesn't affect release/beta, because we use the same bouncer product name (and better file hierarchy).
See Also: → 1246385
@ Matt, can you please take a look at this installer bug? We'll need to uplift the fix to Aurora 53.

(In reply to Rail Aliiev [:rail] ⌚️ET from comment #4)
> To be honest, I would prefer not to have special logic for this in bouncer.
> :(

I hear you. I just thought it might be a good sanity check.
Assignee: nobody → mhowell
status-firefox54: ? → affected
Flags: needinfo?(mhowell)
Priority: -- → P2
First, is this not happening for 32-bit builds? Why not? What's different?

I think I have a workaround I could hack in, but for my part, I'd also prefer not to have special logic in the installer to deal with this, because I honestly don't see how the installer is where that logic belongs.
Flags: needinfo?(mhowell)
The issue happening also for 32-bit builds: select in "Version to install" the 32-bit Firefox Developer Edition option -> install it -> the final installed build is the en-US build, not the specific localization build.
Then something is really odd, because the URL that the stub requests for 32-bit hasn't been changed in 3 years, and that patch just swapped the order of the parameters:
https://hg.mozilla.org/mozilla-central/diff/4e9f43beb31f/browser/branding/aurora/branding.nsi
I'm not even sure if we even served the stub installer via localized pages, maybe this is why we just noticed this.

Paul, do you know if we serve localized stub installers for nightly/aurora via Bedrock?
Flags: needinfo?(pmac)
(In reply to Rail Aliiev [:rail] ⌚️ET from comment #10)
> Paul, do you know if we serve localized stub installers for nightly/aurora
> via Bedrock?

bug 1320662, https://www.mozilla.org/ja/firefox/channel/desktop/#nightly.
Flags: needinfo?(pmac)
We did indeed very recently (Feb 10) start serving localized stub installers for aurora and nightly.

https://github.com/mozilla/bedrock/pull/4621/
Can we fix those stub installers or bouncer quickly, or should I push a patch to the website to disable non-en-US stub installers for nightly and aurora?
I'd revert the change until we have some solution for bouncer. ATM we have to have 2 different products (firefox-aurora-latest-l10n and firefox-aurora-latest) to serve files from different locations.

I see 3 possible solutions here:

1) publish en-US builds to https://archive.mozilla.org/pub/firefox/nightly/latest-mozilla-aurora/ and https://archive.mozilla.org/pub/firefox/nightly/latest-mozilla-aurora-l10n/. Unify the bouncer product names and use https://archive.mozilla.org/pub/firefox/nightly/latest-mozilla-aurora-l10n/ to serve all locales. 

I suspect we may need to touch https://github.com/mozilla-services/product-delivery-tools/tree/master/post_upload or pass some extra parameters to make this work.


2) *Hack* the stub installer to use firefox-aurora-latest for en-US and firefox-aurora-latest-l10n otherwise. 

3) *Hack* bouncer to treat en-US and localized builds differently.


Any other ideas?

I'm going to investigate 1) now
(In reply to Rail Aliiev [:rail] ⌚️ET from comment #14)
> 3) *Hack* bouncer to treat en-US and localized builds differently.

I vote for this, it serves the previous stubs and looks more harmless.
I'm +1 on option 3 as well. I see bouncer's job as being the central source for consistency and truth for product delivery. If we could just ask for the "firefox-aurora-latest" or "firefox-aurora-stub" regardless of language I could further simplify the website and the stub installer wouldn't have to get more complex.
CCing :oremj
I don't like 3) as it's the least transparent option. I'd much rather we just have one directory on archive.m.o; eg, put everything in firefox/nightly/latest-mozilla-aurora/, which is a variation of 1). What do you think might break with that approach Rail, other then people's long standing expectations ? :-)
Flags: needinfo?(nthomas)
(In reply to Nick Thomas [:nthomas] from comment #18)
> I don't like 3) as it's the least transparent option. I'd much rather we
> just have one directory on archive.m.o; eg, put everything in
> firefox/nightly/latest-mozilla-aurora/, which is a variation of 1). What do
> you think might break with that approach Rail, other then people's long
> standing expectations ? :-)

We can try and break the world. \o/

Before doing this I'd like to get some feedback from different parties.

tl;dr, we want to stop publishing localized builds to firefox-{nightly,aurora}-latest-l10n and aggregate everything under firefox-{nightly,aurora}-latest. Would this affect anything that you know?
Flags: needinfo?(pmac)
Flags: needinfo?(l10n)
Flags: needinfo?(francesco.lodolo)
Nothing on my personal list of expectations. If we made that change, we should try hard to remove traces of the old -l10n dirs, or possibly symlink/redirect them? Just to avoid that we have folks downloading old builds by following their old habits.

Also looping in Henrik, for fxuitest automation.
Flags: needinfo?(l10n) → needinfo?(hskupin)
On my side I'm not aware of anything that would break by moving builds from one folder to the other. 

I usually point people to the https://www.mozilla.org/firefox/channel/desktop/ page for download (or the dev version of it if locale is not enabled in production).

+1 on symlinking -l10n to the new folder.
Flags: needinfo?(francesco.lodolo)
AFAIK we can't use symlinks to redirect requests starting with http://archive.mozilla.org/pub/firefox/nightly/latest-mozilla-central-l10n/ to http://archive.mozilla.org/pub/firefox/nightly/latest-mozilla-central/ in S3, but we may find a way to setup redirects. Jeremy may have some input here.
Flags: needinfo?(oremj)
Jeremy, also, if we decide to unify the uploads, we'll switch to passing "-b $branch" instead of "-b $branch-l10n" to post_upload. I wonder if we need to address anything in post_upload in that case, for example https://github.com/mozilla-services/product-delivery-tools/blob/109a64873780b9f1854ad93b94e02b3c20d067ea/post_upload/postupload/release.go#L118. We may want to leave it as is because of Thunderbird maybe?..
Let's also NI some TB folks. See comment 19
Flags: needinfo?(vseerror)
Flags: needinfo?(philipp)
To clarify, I was asking to symlink the latest-mozilla-central-l10n folder to latest-mozilla-central, not redirecting download links pointing to files inside that folder. I assume(d) pike was asking the same.
(In reply to Rail Aliiev [:rail] ⌚️ET from comment #19)
> tl;dr, we want to stop publishing localized builds to
> firefox-{nightly,aurora}-latest-l10n and aggregate everything under
> firefox-{nightly,aurora}-latest. Would this affect anything that you know?

This will definitely break the capability of mozdownload to grab those builds from archive.mozilla.org. It would have to be updated to support both locations to keep compatibility for downloading older builds.
Flags: needinfo?(hskupin)
I'm all for this change from a www.m.o perspective. I'll just need a heads-up before the -l10n products are removed from bouncer so that I can push changes to the website.

It also sounds like this might not be the quickest change. I'll go ahead and change the site for now to only serve nightly and aurora stub installers to en-US for a quick fix while we do the Real Fix.
Flags: needinfo?(pmac)
(In reply to Rail Aliiev [:rail] ⌚️ET from comment #22)
> AFAIK we can't use symlinks to redirect requests starting with
> http://archive.mozilla.org/pub/firefox/nightly/latest-mozilla-central-l10n/
> to http://archive.mozilla.org/pub/firefox/nightly/latest-mozilla-central/ in
> S3, but we may find a way to setup redirects. Jeremy may have some input
> here.

Can we leave old files where they lay and add an index file to -l10n that states it is a deprecated path? Avoiding redirects is preferable.
Flags: needinfo?(oremj)
(In reply to Jeremy Orem [:oremj] from comment #29)
> Can we leave old files where they lay and add an index file to -l10n that
> states it is a deprecated path? Avoiding redirects is preferable.

I'd prefer this too.
Commits pushed to master at https://github.com/mozilla/bedrock

https://github.com/mozilla/bedrock/commit/3f47cd8fa5172de7f25ab13f8d7a06e1a3e03f96
Bug 1339870: Only offer stub for en-US aurora and nightly

Bug 1339870 will fix it such that we can simply use e.g.
firefox-aurora-stub for all locales like we can for beta
and release. So we'll be able to remove our special handling of
nightly and aurora. Until then we have to only serve stub to
en-US windows users.

https://github.com/mozilla/bedrock/commit/5ab2a7df5d1d5f03e79c7b04c48b69559edacca4
Merge pull request #4673 from pmac/nightly-aurora-stub-en-US-only-1339870

Bug 1339870: Only offer stub for en-US aurora and nightly
(In reply to Rail Aliiev [:rail] ⌚️ET from comment #24)
> Let's also NI some TB folks. See comment 19

I can't think of anything. But I don't operate in this space. Perhaps Theo will know.
Flags: needinfo?(vseerror) → needinfo?(theo)
(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #32)
> (In reply to Rail Aliiev [:rail] ⌚️ET from comment #24)
> > Let's also NI some TB folks. See comment 19
> 
> I can't think of anything. But I don't operate in this space. Perhaps Theo
> will know.

As long as the changes are applied to download links on mozilla.org for Thunderbird, I can’t think of anything that could be broken
Flags: needinfo?(theo)
I'm not actually working on this; it seems like Rail is the last person who was, but I can't tell for sure because things seem to have stalled.
Assignee: mhowell → rail
Chris,

How bad does this affect the project? We talked about 1) in comment 14 and it doesn't sound like something we can resolve quickly, because it'll break many many systems. :/

We are left with 2 other options (modify the stub installer or modify bouncer), unless we have something better in mind.
Flags: needinfo?(cpeterson)
(In reply to Rail Aliiev [:rail] ⌚️ET from comment #35)
> How bad does this affect the project? We talked about 1) in comment 14 and
> it doesn't sound like something we can resolve quickly, because it'll break
> many many systems. :/
> 
> We are left with 2 other options (modify the stub installer or modify
> bouncer), unless we have something better in mind.

The Firefox 53 stub installer will include 64-bit as a non-default option, so ideally we would have a solution for localized 64-bit builds before 53 is released (in six weeks on April 18).

@ Rail: assuming we won't have the preferred solution (unifying the latest-mozilla-$CHANNEL and latest-mozilla-$CHANNEL-l10n directories) before April 18, do you recommend that we hack the stub installer or bouncer? Based on comment 15 and comment 16, it sounds like teaching bouncer about the two different directories is the preferred temporary solution.
Flags: needinfo?(cpeterson) → needinfo?(rail)
Resummarizing for clarity - affects nightly and aurora only because of the way builds are uploaded, but not beta or release. It's both flavors of windows rather than just Win64.
Summary: Win 64 stub installer on aurora defaults to installing en-us → Localised stub installer for Aurora and Nightly installs en-US Firefox
Thanks for the clarification, Nick! Since this problem affects both Win32 and Win64 and only affects Nightly and Aurora channels, this bug doesn't need to block our Win64 rollout to release.
No longer blocks: support-win64, win64-rollout
Flags: needinfo?(rail)
Flags: needinfo?(philipp)
Assignee: rail → nobody
Component: Other → General Automation
Priority: P2 → P3
QA Contact: mshal → catlee
Moving the tracking flags to 55, as 53 is currently Beta and 54 will soon go to Beta.
status-firefox53: affected → unaffected
status-firefox55: --- → affected
tracking-firefox55: --- → +
Rail, is someone else going to work on this for the 55 timeframe?
status-firefox54: affected → unaffected
Flags: needinfo?(rail)
I expect it to be fixed for Dev Edition this cycle. No changes for Nightly unfortunately. :/
Flags: needinfo?(rail)
Hi Chris, is this something we can fix soon? I think leaving this bug lingering on Nightly should also be a cause for concern though it doesn't block the feature's merge to Beta as Chris mentioned in comment 38. Wdyt?
Flags: needinfo?(catlee)
Priority: P3 → P2
Summary: Localised stub installer for Aurora and Nightly installs en-US Firefox → Localised stub installer for Nightly installs en-US Firefox
Yes, we can start looking at this after the rest of the Dawn projects are wrapped up.
Flags: needinfo?(catlee)
I'm going to tackle this soon.
Assignee: nobody → rail
Priority: P2 → P1
Discussed with Rail. I'm starting to work on this bug.
Assignee: rail → jlorenzo
Status: NEW → ASSIGNED
Hey :oremj!

I discussed a potential solution with Rail yesterday. It seems post_upload is the way to go. From what I understood, it's just a matter of returning 1 more "dest", in the case of nightly-en-US. Does that sound correct to you? 

I tested the patch with a new unit test. I haven't tried it against a real instance. If that's needed, I don't mind using a staging S3 bucket that releng has.

By the way, this is my first attempt at Go. So I may be clumsy on the specificity of this language. Don't hesitate to point out nits and bigger errors!
Attachment #8873499 - Flags: review?(oremj)
Does https://github.com/mozilla-services/product-delivery-tools/pull/49 look okay to you? I've made a few small changes.
Flags: needinfo?(jlorenzo)
Yes, the changes look good to me. Thank you for your rapid answer!
Flags: needinfo?(jlorenzo)
Thanks for merging #49, :oremj. I guess the next step is to release a new version of product-delivery-tools and deploy it on production. Is that correct?
Flags: needinfo?(oremj)
Depends on: 1372284
Discussed over IRC. Deployment of comment 47 will be done in bug 1372284.
Flags: needinfo?(oremj)
Attachment #8873499 - Flags: review?(oremj) → review+
We had to back the deploy out because of bug 1372461.
Capturing IRC debugging of the problem:

<oremj> ah ha: https://github.com/aws/aws-sdk-go/issues/1058
<oremj> I need to update the sdk, because there is a bug against go 1.8
Host with sdk upgrade has been promoted.
Blocks: 1373673
The promotion worked. Thank you very much :oremj!

I changed the rules for firefox-nightly-latest[1]. Now win, win64 and mac point to the l10n folder passing the real locale. I tested these direct links:
* http://download.mozilla.org/?os=osx&lang=es-ES&product=firefox-nightly-latest
* http://download.mozilla.org/?os=win&lang=es-ES&product=firefox-nightly-latest
* http://download.mozilla.org/?os=win63&lang=es-ES&product=firefox-nightly-latest

They allowed me to download the right installers. I also tested the es-ES stub installers for both win and win64. In each case, I got the correct locale installed.

The next step is to move linux builds to the l10n folder. But let's handle that in bug 1373673. 

[1] https://bounceradmin.mozilla.com/admin/mirror/location/?product__id__exact=2005
Status: ASSIGNED → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
For the record, I changed the locations to:
> /firefox/nightly/latest-mozilla-central-l10n/firefox-56.0a1.:lang.win32.installer.exe

This is possible because the bouncer performs a simple string substitution in the URL[1]

[1] https://github.com/mozilla/tuxedo/blob/1269002acef613f7a02726379aa980f6156c5ab3/bouncer/php/index.php#L160
One thing I just realized while talking to :RT, stub links like [1] weren't localized. I just updated firefox-nightly-stub[2] to point to the same locations as firefox-nightly-stub-l10n[3].

I tested [1] again and got the localized stub.

[1] https://download.mozilla.org/?product=firefox-nightly-stub&os=win&lang=fr
[2] https://bounceradmin.mozilla.com/admin/mirror/location/?product__id__exact=6509
[3] https://bounceradmin.mozilla.com/admin/mirror/location/?product__id__exact=6512
The website was discussed early in this bug. In comment #28 I mentioned that I had to change the site to only serve stub to en-US since stub was giving people en-US regardless of the stub locale. If it's now fixed we can file a bug to update the download buttons to once again serve stub installer to all locales. This is only for Nightly so far though right; we can't yet serve localized stub to beta or release channels yet?
Thanks for bringing this up Paul, I missed that comment. Earlier today, Romain filed bug 1374261 to update the download button.

This bug was indeed for Nightly. However, localized stubs have been available in Beta and Release (I don't know since when). For instance:
* https://download.mozilla.org/?product=firefox-stub&os=win&lang=fr
* https://download.mozilla.org/?product=firefox-beta-stub&os=win&lang=de
Blocks: 1374261
(In reply to Paul [:pmac] McLanahan from comment #57)
> This is only for Nightly so far though right; we can't yet serve
> localized stub to beta or release channels yet?

beta/release/devedtion stubs are fine, we have been serving them for years now afaik.
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: