Closed Bug 1233779 Opened 8 years ago Closed 8 years ago

Don't serve Firefox 43.0.2 and higher to XP SP2 users

Categories

(Release Engineering :: Release Requests, defect)

defect
Not set
normal

Tracking

(firefox43+ affected)

RESOLVED FIXED
Tracking Status
firefox43 + affected

People

(Reporter: rail, Unassigned)

References

Details

Hi,

Sorry for the late heads up. :( In bug 1079858 we are going to switch to SHA2 signed installers, which do not work in XP SP2 (they won't launch silently).

As a work around we need to serve Firefox 43.0.1 to XP SP2 users. With 43.0.1 installed they will be able to upgrade to 43.0.2+ without any major issues.

I think we were going to set up a special site with SHA1-HTTPS, which can probably used for this purpose.

Holler if I described the issue unclear.
Quick question: how long would we have to do this XP SP2 exception? Estimate?
Paul - am I correct this is a change for bouncer not mozilla.org (afaik we dont want mozilla.org to have this knowledge). Can you confirm and if so, route to the right bouncer person.
Flags: needinfo?(pmac)
Hi Travis, Jeremy; Laura Thomson suggested I should also let you all know about this bug and the work we need to do to support XP users through next year. 

Chris, I think I've read some projections of XP sp2 users that have them tapering off to almost no one by mid-2017. From my perspective, our doing this supports Firefox users in developing countries (where the bulk of XP users are concentrated). We should think about ESR timelines here too.
Flags: needinfo?(tblow)
Flags: needinfo?(oremj)
(In reply to Ben (:bensternthal) from comment #2)
> Paul - am I correct this is a change for bouncer not mozilla.org (afaik we
> dont want mozilla.org to have this knowledge). Can you confirm and if so,
> route to the right bouncer person.

I don't think bouncer can detect XP SP2.
(In reply to Chris More [:cmore] from comment #1)
> Quick question: how long would we have to do this XP SP2 exception? Estimate?

Forever? :) Or maybe until our SHA1 website cert is expired? I'm just guessing, probably better to ask someone else.
Tracking for 43. 
Just to be clear. We should hold back the 43.0.2 release (planned for next Tuesday) until this is in place? Or no?
(In reply to Rail Aliiev [:rail] from comment #4)
> (In reply to Ben (:bensternthal) from comment #2)
> > Paul - am I correct this is a change for bouncer not mozilla.org (afaik we
> > dont want mozilla.org to have this knowledge). Can you confirm and if so,
> > route to the right bouncer person.
> 
> I don't think bouncer can detect XP SP2.

I'm not sure whether bedrock can reliably either. We can pretty easily pin XP users to 43.0.1 I think. I'll see what I can find out.
FYI: I believe that most XP SP2 users have "SV1" in their user agent string.
(In reply to Chris More [:cmore] from comment #8)
> FYI: I believe that most XP SP2 users have "SV1" in their user agent string.

For most browsers? Problem is that this is OS specific and it's very difficult (if possible at all) to reliably detect an exact OS version like this. Not to mention that this is in effect a new platform for our buttons, so doing this at all will be significant effort for implementation and especially testing. And this is over a very heavy PTO time. It will likely take a few weeks to get this out the door.

Ben: I think I failed to answer you from comment #2. I do think this is the domain of www.mozilla.org. Bouncer normally just serves up the version, platform, and locale that we tell it. I think it falls to us to tell it 43.0.1 for these users unfortunately. But like I said above, this will be unreliable at best.
Flags: needinfo?(pmac)
(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #6)
> Tracking for 43. 
> Just to be clear. We should hold back the 43.0.2 release (planned for next
> Tuesday) until this is in place? Or no?

AFAIK, 43.0.2 won't launch on Windows XP SP2.
(In reply to Paul [:pmac] McLanahan from comment #9)
> (In reply to Chris More [:cmore] from comment #8)
> > FYI: I believe that most XP SP2 users have "SV1" in their user agent string.
> 
> For most browsers? Problem is that this is OS specific and it's very
> difficult (if possible at all) to reliably detect an exact OS version like
> this. Not to mention that this is in effect a new platform for our buttons,
> so doing this at all will be significant effort for implementation and
> especially testing. And this is over a very heavy PTO time. It will likely
> take a few weeks to get this out the door.

+1 -- it can be hard, unreliable, and fragile. Just wanted to mention that my previous research on how find SP2 showed that there is some signal, but I would be concerned how reliable it is. Not a solution, just a statement.

> 
> Ben: I think I failed to answer you from comment #2. I do think this is the
> domain of www.mozilla.org. Bouncer normally just serves up the version,
> platform, and locale that we tell it. I think it falls to us to tell it
> 43.0.1 for these users unfortunately. But like I said above, this will be
> unreliable at best.
We can add a hack to bouncer that basically does:

if Windows NT 5.1 in UserAgent:
    serve sha1 signed 43.0.0

I need to know:
* How soon do we need this?

* Should we do this in bouncer?

* Is it okay to capture all Windows XP users, we can't detect service packs?

* If we are doing it, which product should I serve?
Flags: needinfo?(oremj)
(In reply to Jeremy Orem [:oremj] from comment #12)
> We can add a hack to bouncer that basically does:
> 
> if Windows NT 5.1 in UserAgent:
>     serve sha1 signed 43.0.0
> 
> I need to know:
> * How soon do we need this?
> 
> * Should we do this in bouncer?
> 
> * Is it okay to capture all Windows XP users, we can't detect service packs?
> 
> * If we are doing it, which product should I serve?

pmac brought up the caching issue, which I hadn't considered. We'd have to Vary on User-Agent, which would destroy caching, making this much tougher to pull off on the bouncer side.
As soon as possible before Jan. 1. As we want as many users to get the updates before January 1 and we need time for testing.     If we give this to all XP users now, then fine tune the sp 2 detection later, that might work. 


(In reply to Jeremy Orem [:oremj] from comment #12)
> We can add a hack to bouncer that basically does:
> 
> if Windows NT 5.1 in UserAgent:
>     serve sha1 signed 43.0.0
> 
> I need to know:
> * How soon do we need this?
> 
> * Should we do this in bouncer?
> 
> * Is it okay to capture all Windows XP users, we can't detect service packs?
> 
> * If we are doing it, which product should I serve?
The current plan:

We discussed this and discovered that the 43.0.1 release that is happening now is sufficient through the end of the year. Early 2016 43.0.2 will be released and we must have a solution in bedrock in place before then since WinXP SP3 users are the only XP users that will be able to install 43.0.2+. This affects all of the channels, so we'll need to pin XP SP2 users to the versions of those channels (nightly will have to be handled separately since that's a separate site) that are currently released. The first iteration of the fix for bedrock will just be to pin all XP users to the currently released versions. We'll be filing another bug to make sure that we're only pinning XP users that aren't on SP3 before the release of Fx 44.

Lemme' sum-up:

1. Pin all XP users to currently released versions of all channels (release, beta, dev) before the end of the year.

2. Dial back that pinning as much as we can to only target XP users _not_ on SP3 before the release of Fx 44.

That's my current understanding. Please add to that as you're able. We'll get started on implementation planning on the bedrock side for #1 now. For #2 it's possible that we could work with :oremj to send bouncer an indication that the user is XP and let it reduce caching for just those users and dial back the version pinning on the bouncer side. But that will be a discussion in the bug for #2, which I'll be filing shortly and will be blocked by this one.
> 2. Dial back that pinning as much as we can to only target XP users _not_ on SP3 before the release of Fx 44.

We know XPSP2 users will be connecting to Cloudflare using SSLv3. I _think_ XPSP3 users will connect with TLSv1. If true, you could ask Cloudflare to provide a header with the TLS version used, you use information to differentiate between users of XP SP2 and 3.
>> Quick question: how long would we have to do this XP SP2 exception? Estimate?
> Forever? :) Or maybe until our SHA1 website cert is expired? I'm just guessing, probably better to ask someone else.

At least for all of 2016. Maybe some of 2017. Our current cert expires Dec.29th 2016, but I suspect there will be other ways to obtain SHA-1 certs after that date.
After meeting with :pmac and :jgmize, current totally not concrete plan for first version of fix is:

- Write JS function to target Windows XP users. Appears checking for "Windows NT 5.1" in navigator.userAgent will work, though will not detect service pack. This function will add "winxp" class to the <html> element. When this class is present, normal download buttons will be hidden.

- Modify download_buttons.html to include alternate, XP only top level element. This element will only be displayed when the <html> element has the "winxp" class applied. This element will contain a button/link to an XP specific version of Firefox. Localized download links will (hopefully) be handled by a combination of predictable link/installer file name structure and locale specified in address bar. (*waves hands*)
(In reply to Julien Vehent [:ulfr] from comment #17)
> > 2. Dial back that pinning as much as we can to only target XP users _not_ on SP3 before the release of Fx 44.
> 
> We know XPSP2 users will be connecting to Cloudflare using SSLv3. I _think_
> XPSP3 users will connect with TLSv1. If true, you could ask Cloudflare to
> provide a header with the TLS version used, you use information to
> differentiate between users of XP SP2 and 3.

We thought about that too, but that will only work for some users right? Some browsers on XP SP2 (e.g. Firefox) will use the TLSv1, and we still need to make sure that those users don't get any build beyond 43.0.1.
You're correct; the protocol detection technique only works to differentiate IE users on XPSP2/3, and would probably need to check user agents as well.

It's a long shot, but maybe cloudflare has some user-agent fingerprinting technique that could help?
It turns out that we can alter bouncer caching based on user agent, so I'm also going to investigate doing this on the bouncer level.
:oremj - Any update on handling things at the bouncer level?
I've been working on this at https://github.com/mozilla-services/go-bouncer/pull/16

I'd like someone to double check the detection logic and that I'm adjusting the product correctly for XP users.
Flags: needinfo?(tblow)
XP users should be pointed to 
nightly: 46 from (sometime last week - we need to pin down to before patch 2 landed)  
aurora: 45 release from today
beta: 44 beta 1 
release: 43.0.1
Rail already figured out the versions we need for most channels: https://bugzilla.mozilla.org/show_bug.cgi?id=1234273   

And we should add the esr38.5.1 version that will build today, I think.
Depends on: 1234273
I've deployed https://github.com/mozilla-services/go-bouncer/pull/16 and an nginx cache change to https://bouncer-bouncer.stage.mozaws.net/. I also synchronized the stage database with production.
Liz, do we need to pin ESR to the currently released version?
Flags: needinfo?(lhenry)
Yes, that makes sense (for XP users). Thank you Jeremy!
Flags: needinfo?(lhenry)
I should specify the version -- 38.5.1esr.
I've added ESR detection to bouncer and deployed to stage.
The bouncer changes have been deployed to download.mozilla.org.
I tried to verify Beta with one failure. Looks like the patch expects "firefox-beta", while the actual product name is "firefox-beta-latest", see https://bounceradmin.mozilla.com/admin/mirror/productalias/

It worked fine for Firefox-beta-stub.

These are the curl commands I used:


# XP, Stub, PASS
$ curl -s -I -A "Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0" "http://download.mozilla.org/?product=Firefox-beta-stub&os=win&lang=en-US" | grep -i location
Location: https://download-installer.cdn.mozilla.net/pub/firefox/releases/44.0b1/win32/en-US/Firefox%20Setup%20Stub%2044.0b1.exe

# Not XP, Stub, PASS
$ curl -s -I -A "Mozilla/5.0 (Windows NT 6.1; rv:31.0) Gecko/20100101 Firefox/31.0" "http://download.mozilla.org/?product=Firefox-beta-stub&os=win&lang=en-US" | grep -i location
Location: https://download-installer.cdn.mozilla.net/pub/firefox/releases/44.0b2/win32/en-US/Firefox%20Setup%20Stub%2044.0b2.exe

# XP, Full, FAIL
$ curl -s -I -A "Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0" "http://download.mozilla.org/?product=firefox-beta-latest&os=win&lang=en-US" | grep -i location               
Location: http://download.cdn.mozilla.net/pub/firefox/releases/44.0b2/win32/en-US/Firefox%20Setup%2044.0b2.exe

# Not XP, Full, PASS
$ curl -s -I -A "Mozilla/5.0 (Windows NT 6.1; rv:31.0) Gecko/20100101 Firefox/31.0" "http://download.mozilla.org/?product=firefox-beta-latest&os=win&lang=en-US" | grep -i location
Location: http://download.cdn.mozilla.net/pub/firefox/releases/44.0b2/win32/en-US/Firefox%20Setup%2044.0b2.exe
BTW, does https://github.com/mozilla-services/go-bouncer/pull/16/files#diff-31a4a7824225629df071a1cba8b785d5R29 mean that we serve 38.5.2esr to XP users? it should be 38.5.1esr per comment #30.
Just checked 43.0.x, it works as expected.

$ curl -s -I -A "Mozilla/5.0 (Windows NT 6.1; rv:31.0) Gecko/20100101 Firefox/31.0" "http://download.mozilla.org/?product=firefox-latest&os=win&lang=en-US" | grep -i location
Location: http://download.cdn.mozilla.net/pub/firefox/releases/43.0.2/win32/en-US/Firefox%20Setup%2043.0.2.exe

$ curl -s -I -A "Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0" "http://download.mozilla.org/?product=firefox-latest&os=win&lang=en-US" | grep -i location
Location: http://download.cdn.mozilla.net/pub/firefox/releases/43.0.1/win32/en-US/Firefox%20Setup%2043.0.1.exe
Most of the beta links on the site look like this: 

https://download.mozilla.org/?product=firefox-44.0b2-SSL&os=win&lang=en-US

We'd need that to return "44.0b1", but so far it does not:

$ curl -s -I -A "Mozilla/5.0 (Windows NT 6.1; rv:31.0) Gecko/20100101 Firefox/31.0" "https://download.mozilla.org/?product=firefox-44.0b2-SSL&os=win&lang=en-US" | grep -i location
Location: https://download-installer.cdn.mozilla.net/pub/firefox/releases/44.0b2/win32/en-US/Firefox%20Setup%2044.0b2.exe
HA! Of course I noticed my mistake right after hitting submit :( It does work if I use the right UA string:

$ curl -s -I -A "Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0" "https://download.mozilla.org/?product=firefox-44.0b2-SSL&os=win&lang=en-US" | grep -i location
Location: https://download-installer.cdn.mozilla.net/pub/firefox/releases/44.0b1/win32/en-US/Firefox%20Setup%2044.0b1.exe

Please ignore comment #36
I've deployed the beta/ESR fixes to stage and production.
beta links LGTM!

Let's declare victory! Thanks all!
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
While this specific issue is resolved, we cannot yet declare victory on the larger problem: see bug 1234874 and bug 1234882 for more details.
(In reply to Josh Mize [:jgmize] from comment #40)
> While this specific issue is resolved, we cannot yet declare victory on the
> larger problem: see bug 1234874 and bug 1234882 for more details.

Let's address those issues in separate bugs maybe?
See Also: → 1240511
Changing the product/component as this was solved in the bouncer.
Component: Bedrock → Releases
Product: www.mozilla.org → Release Engineering
QA Contact: rail
Version: Production → unspecified
You need to log in before you can comment on or make changes to this bug.