Closed Bug 1079858 Opened 10 years ago Closed 8 years ago

Deal with deprecation of SHA1 (SHA-1) Authenticode signatures for Windows signing

Categories

(Release Engineering :: Release Requests, defect)

Product:
Release Engineering
Component:
Release Requests
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(firefox42 wontfix, firefox43+ fixed, firefox44+ fixed, firefox45 fixed, firefox-esr3844+ fixed, relnote-firefox 43+)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox42 --- wontfix
firefox43 + fixed
firefox44 + fixed
firefox45 --- fixed
firefox-esr38 44+ fixed
relnote-firefox --- 43+

People

(Reporter: bhearsum, Assigned: molly)

References

(
URL
)

Details

Attachments

(4 files, 5 obsolete files)

dual signing on Win7 with update installed
12.15 KB, image/png
Details
untested patch attempt for partner repacks
57 bytes, text/x-github-pull-request
Details | Review
Part 2: Use SHA-2 to sign Windows binaries, and switch the stub installer to expect that
7.26 KB, patch
ritu
: approval-mozilla-beta+
ritu
: approval-mozilla-release+
Sylvestre
: approval-mozilla-esr38+
nthomas
: checked-in+
Details | Diff | Splinter Review
Part 1: Inform the updater and maintenance service of the new SHA2 certicicate issuer
5.32 KB, patch
robert.strong.bugs
: review+
ritu
: approval-mozilla-beta+
ritu
: approval-mozilla-release+
ritu
: approval-mozilla-esr38+
nthomas
: checked-in+
Details | Diff | Splinter Review 
Gerv informed me of Microsoft's plans to stop support SHA1 signatures (and possibly certificates) on 2016-01-01. The wrinkle here is that XP SP2 does _not_ support sha2 or other newer signature algorithms. This means that if we need to ship to XP SP2 _and_ XP SP3 (and higher) after 2016-01-01, we may not be able to ship them the same binaries. Some potential workarounds:
* Look into the possibility of multiple signatures or one signature with multiple algorithms on the same file. This is the best solution if it's possible IMO - it shouldn't be a ton of work.
* Sign our binaries with two different certs (sha1 for XP SP2, sha2 or higher for XP SP3 and up). This option is the most amount of work, and will have us storing significantly more bits per release. It probably also makes our CDN bill go up.
* Serve an older build to XP SP2 users from our website and let them update to the latest after install (this will still work because the timestamp on the signature will be older than 2016-01-01). This is a crappy user experience, but easier than shipping two builds.

If we drop support for XP SP2 before 2016-01-01, we don't need to do any of the above, but we do need to buy a new certificate from a non-sha1 root. I don't think we'll have any trouble getting this from our preferred CA (Digicert).

I couldn't figure out how to get XP usage numbers by service pack, but if the number is already small, it seems like dropping support before 2016 is the way to go.
Updating summary, because regardless of what happens with XP SP2, we need to buy a new cert.
Summary: deal with deprecation of sha1 authenticode signatures (if we haven't dropped support for XP SP2 by 2016-01-01) → deal with deprecation of sha1 authenticode signatures
(Make it a little easier to find in 6 months.)

Gerv
Summary: deal with deprecation of sha1 authenticode signatures → Deal with deprecation of SHA1 (SHA-1) Authenticode signatures for Windows signing
Vista requires an update too: http://support.microsoft.com/kb/2763674
I think Vista marketshare is relatively small right? I think a warning on the download page should be good enough.
There is also an update for Windows 7, which MS has now tried and failed twice to issue:
http://www.theregister.co.uk/2015/03/13/microsoft_reborks_latest_windows_7_patch/
https://technet.microsoft.com/en-us/library/security/3033929.aspx

This is concerning because I bet a non-zero number of clients of these various OSes may miss out on this update. If it were just old XP, that would be one thing, but it seems to be Vista and 7 as well.

The deadline for this is 9 months away, which is not long. What's the plan to make a plan, here?

Gerv
Win7 supports SHA2 certs fine for Authenticode signing without this patch.
To clarify, "Authenticode signing" refers to user mode only. Mozilla products don't ship with any kernel mode drivers.
As a last resort, note that from http://blogs.technet.com/b/pki/archive/2013/11/12/sha1-deprecation-policy.aspx :
"Update: For code signing certificates, CAs may continue issuing new SHA1 code signing certificates to ensure that developers targeting Windows Vista and Windows Server 2008 are properly supported."
And you can then use signtool /fd sha1 with the SHA1 cert and then /fd sha256 with the SHA256 cert.
(In reply to Gervase Markham [:gerv] from comment #5)
> There is also an update for Windows 7, which MS has now tried and failed
> twice to issue:
> http://www.theregister.co.uk/2015/03/13/
> microsoft_reborks_latest_windows_7_patch/
> https://technet.microsoft.com/en-us/library/security/3033929.aspx
> 
> This is concerning because I bet a non-zero number of clients of these
> various OSes may miss out on this update. If it were just old XP, that would
> be one thing, but it seems to be Vista and 7 as well.
> 
> The deadline for this is 9 months away, which is not long. What's the plan
> to make a plan, here?
> 
> Gerv

I'll bring it up with Release Management sometime next month. Assigning to me for now, not sure if I'll end up doing actual work here though.
Assignee: nobody → bhearsum
EricLaw has test files available at http://www.enhanceie.com/test/authenticode.htm (all signed with a SHA256 certificate)
(In reply to Ben Hearsum [:bhearsum] from comment #0)
> * Look into the possibility of multiple signatures or one signature with
> multiple algorithms on the same file. This is the best solution if it's
> possible IMO - it shouldn't be a ton of work.

Looks like this isn't possible. Based on https://github.com/mkrautz/osslsigncode/blob/master/osslsigncode.c#L2660, it appears that multiple signatures are only supported when each uses the same hash type :(.

> * Sign our binaries with two different certs (sha1 for XP SP2, sha2 or
> higher for XP SP3 and up). This option is the most amount of work, and will
> have us storing significantly more bits per release. It probably also makes
> our CDN bill go up.

> * Serve an older build to XP SP2 users from our website and let them update
> to the latest after install (this will still work because the timestamp on
> the signature will be older than 2016-01-01). This is a crappy user
> experience, but easier than shipping two builds.

Another option is to deprecated SP2. I filed bug 1178797 to get some data on XP SP2 vs. SP3 usage, which should help us make a more informed decision here.
(In reply to Ben Hearsum [:bhearsum] from comment #11)
> (In reply to Ben Hearsum [:bhearsum] from comment #0)
> > * Look into the possibility of multiple signatures or one signature with
> > multiple algorithms on the same file. This is the best solution if it's
> > possible IMO - it shouldn't be a ton of work.
> 
> Looks like this isn't possible. Based on
> https://github.com/mkrautz/osslsigncode/blob/master/osslsigncode.c#L2660, it
> appears that multiple signatures are only supported when each uses the same
> hash type :(.

It is possible if you use MS's signtool.

> > * Sign our binaries with two different certs (sha1 for XP SP2, sha2 or
> > higher for XP SP3 and up). This option is the most amount of work, and will
> > have us storing significantly more bits per release. It probably also makes
> > our CDN bill go up.
> 
> > * Serve an older build to XP SP2 users from our website and let them update
> > to the latest after install (this will still work because the timestamp on
> > the signature will be older than 2016-01-01). This is a **** user
> > experience, but easier than shipping two builds.
> 
> Another option is to deprecated SP2. I filed bug 1178797 to get some data on
> XP SP2 vs. SP3 usage, which should help us make a more informed decision
> here.

Yea, there is also the matter of the browser download site also needing to move to SHA2 certs anyway.
(In reply to Yuhong Bao from comment #12)
> (In reply to Ben Hearsum [:bhearsum] from comment #11)
> > (In reply to Ben Hearsum [:bhearsum] from comment #0)
> > > * Look into the possibility of multiple signatures or one signature with
> > > multiple algorithms on the same file. This is the best solution if it's
> > > possible IMO - it shouldn't be a ton of work.
> > 
> > Looks like this isn't possible. Based on
> > https://github.com/mkrautz/osslsigncode/blob/master/osslsigncode.c#L2660, it
> > appears that multiple signatures are only supported when each uses the same
> > hash type :(.
> 
> It is possible if you use MS's signtool.

Which is not an option for us, unfortunately. We have significant investment in Linux-based signing infrastructure.

> > > * Sign our binaries with two different certs (sha1 for XP SP2, sha2 or
> > > higher for XP SP3 and up). This option is the most amount of work, and will
> > > have us storing significantly more bits per release. It probably also makes
> > > our CDN bill go up.
> > 
> > > * Serve an older build to XP SP2 users from our website and let them update
> > > to the latest after install (this will still work because the timestamp on
> > > the signature will be older than 2016-01-01). This is a **** user
> > > experience, but easier than shipping two builds.
> > 
> > Another option is to deprecated SP2. I filed bug 1178797 to get some data on
> > XP SP2 vs. SP3 usage, which should help us make a more informed decision
> > here.
> 
> Yea, there is also the matter of the browser download site also needing to
> move to SHA2 certs anyway.

This should be a separate bug, probably against www.mozilla.org. Should be pretty simple in comparison though, just needs a new SSL certificate if I understand correctly.
> > * Sign our binaries with two different certs (sha1 for XP SP2, sha2 or
> > higher for XP SP3 and up). This option is the most amount of work, and will
> > have us storing significantly more bits per release. It probably also makes
> > our CDN bill go up.
> 
> > * Serve an older build to XP SP2 users from our website and let them update
> > to the latest after install (this will still work because the timestamp on
> > the signature will be older than 2016-01-01). This is a **** user
> > experience, but easier than shipping two builds.
> 
> Another option is to deprecated SP2. I filed bug 1178797 to get some data on
> XP SP2 vs. SP3 usage, which should help us make a more informed decision
> here.

BTW, this bug is closed, and I am interested in the data.
(In reply to Yuhong Bao from comment #14)
> > > * Sign our binaries with two different certs (sha1 for XP SP2, sha2 or
> > > higher for XP SP3 and up). This option is the most amount of work, and will
> > > have us storing significantly more bits per release. It probably also makes
> > > our CDN bill go up.
> > 
> > > * Serve an older build to XP SP2 users from our website and let them update
> > > to the latest after install (this will still work because the timestamp on
> > > the signature will be older than 2016-01-01). This is a **** user
> > > experience, but easier than shipping two builds.
> > 
> > Another option is to deprecated SP2. I filed bug 1178797 to get some data on
> > XP SP2 vs. SP3 usage, which should help us make a more informed decision
> > here.
> 
> BTW, this bug is closed, and I am interested in the data.

There's nothing in it yet and I'm not sure how public we make our ADI data either...I'll see what I can do once we have the data.
(In reply to Ben Hearsum [:bhearsum] from comment #15)
> (In reply to Yuhong Bao from comment #14)
> > > > * Sign our binaries with two different certs (sha1 for XP SP2, sha2 or
> > > > higher for XP SP3 and up). This option is the most amount of work, and will
> > > > have us storing significantly more bits per release. It probably also makes
> > > > our CDN bill go up.
> > > 
> > > > * Serve an older build to XP SP2 users from our website and let them update
> > > > to the latest after install (this will still work because the timestamp on
> > > > the signature will be older than 2016-01-01). This is a **** user
> > > > experience, but easier than shipping two builds.
> > > 
> > > Another option is to deprecated SP2. I filed bug 1178797 to get some data on
> > > XP SP2 vs. SP3 usage, which should help us make a more informed decision
> > > here.
> > 
> > BTW, this bug is closed, and I am interested in the data.
> 
> There's nothing in it yet and I'm not sure how public we make our ADI data
> either...I'll see what I can do once we have the data.

I am particularly interested in the market share of both XP SP2 and XP SP3.
(In reply to Ben Hearsum [:bhearsum] from comment #13)
> > > > * Sign our binaries with two different certs (sha1 for XP SP2, sha2 or
> > > > higher for XP SP3 and up). This option is the most amount of work, and will
> > > > have us storing significantly more bits per release. It probably also makes
> > > > our CDN bill go up.
> > > 
> > > > * Serve an older build to XP SP2 users from our website and let them update
> > > > to the latest after install (this will still work because the timestamp on
> > > > the signature will be older than 2016-01-01). This is a **** user
> > > > experience, but easier than shipping two builds.
> > > 
> > > Another option is to deprecated SP2. I filed bug 1178797 to get some data on
> > > XP SP2 vs. SP3 usage, which should help us make a more informed decision
> > > here.
> > 
> > Yea, there is also the matter of the browser download site also needing to
> > move to SHA2 certs anyway.
> 
> This should be a separate bug, probably against www.mozilla.org. Should be
> pretty simple in comparison though, just needs a new SSL certificate if I
> understand correctly.

I filed bug 1178952 for this. Turns out we also need to upgrade download.mozilla.org's cert, too.
A few of us have been talking out of band about this, I'll be coming back to this bug with a plan in a week or two.

I did some more research into this and things are more complicated than I thought. According to the comments on http://blogs.technet.com/b/pki/archive/2013/11/12/sha1-deprecation-policy.aspx and the table on https://www.globalsign.com/en/blog/microsoft-announces-updates-sha-1-code-signing-policy/, there is no one signature algorithm that will be supported by XP, Vista, Windows 7, Windows 8, and Windows 10 on January 1, 2016. Once we hit that date, XP and Vista will only support SHA1, while Windows 7 and up will only support SHA2.

Unless I've interpreted things wrong, this means that we need to do both SHA1 and SHA2 signing in some way. Hopefully we can find a way to do multiple signatures on the same binary rather than shipping two different sets of binaries.
SHA2 certificates != SHA2 signatures
I talked with Yuhong a bunch yesterday to try and get more up to speed. If I understood correctly, up to date versions of XP and Vista should work fine with a SHA2 certificate (for XP this means SP3, for Vista this means a particular KB). Windows 7, 8, and 10 should be fine regardless of version. The table that I linked to in my prior comment was actually about kernel driver support, which doesn't affect us. DigiCert has a better table, which agrees with everything that Yuhong told me. The "Authenticode" column of "Code Signing Compatibility" on https://www.digicert.com/sha-2-compatibility.htm is what we care about.

I'm in the process of getting a new SHA2 code signing certificate from DigiCert right now. Once I have that I'll sign something with it and we can verify on the various platforms.

Thanks again to Yuhong for helping me understand compatibility on various platforms.
(In reply to Ben Hearsum [:bhearsum] from comment #20)
> If I understood correctly, up to date versions of XP and Vista should work fine
> with a SHA2 certificate (for XP this means SP3, for Vista this means a
> particular KB).

Just that a huge chunk of XP users never did get SP3 and are still sitting on SP2 even now. The market share numbers would really be interesting there.
OK, we've got some data now. Based on incoming update requests, it looks like 2/3rds of our Windows XP users are on SP3, while 1/3rd are on SP2. I need to find the right set of people to make a call on what this means for deprecation, but RyanVM reminded me today that it's very unlikely that we'll be able to desupport XP SP on ESR38. So even if we do so for mainline Firefox, we probably still need to deal with dual signing to make sure that ESR38 works on XP SP2 & up.
Looks like we're not the first ones to have issues with osslsigncode + nested signatures. https://github.com/mumble-voip/mumble/issues/1308 talks about it, and even points at code that is claimed to work. Unfortunately, I couldn't reproduce their results with our binaries. It's possible that their patchset only works on MSIs...
(In reply to Ben Hearsum [:bhearsum] from comment #23)
> Looks like we're not the first ones to have issues with osslsigncode +
> nested signatures. https://github.com/mumble-voip/mumble/issues/1308 talks
> about it, and even points at code that is claimed to work. Unfortunately, I
> couldn't reproduce their results with our binaries. It's possible that their
> patchset only works on MSIs...

I looked into this some more, including speaking with the one of their RelEng people, and it's looking like only Windows 8 and higher support multiple signatures. When I look at a binary I've signed from Windows 8, I see both signatures. AFAICT, Windows 7 and earlier only see the first signature. If this overlapped with the SHA1-only versions of Windows we might've been able to get away with doing the SHA1 signature first, but since it doesn't we can't tweak signing order to support all platforms in one binary :(.
(In reply to Ben Hearsum [:bhearsum] from comment #24)
> (In reply to Ben Hearsum [:bhearsum] from comment #23)
> > Looks like we're not the first ones to have issues with osslsigncode +
> > nested signatures. https://github.com/mumble-voip/mumble/issues/1308 talks
> > about it, and even points at code that is claimed to work. Unfortunately, I
> > couldn't reproduce their results with our binaries. It's possible that their
> > patchset only works on MSIs...
> 
> I looked into this some more, including speaking with the one of their
> RelEng people, and it's looking like only Windows 8 and higher support
> multiple signatures. When I look at a binary I've signed from Windows 8, I
> see both signatures. AFAICT, Windows 7 and earlier only see the first
> signature. 

Win7 with https://technet.microsoft.com/en-us/library/security/3033929 should support dual signatures too.
(In reply to Yuhong Bao from comment #25)
> (In reply to Ben Hearsum [:bhearsum] from comment #24)
> > (In reply to Ben Hearsum [:bhearsum] from comment #23)
> > > Looks like we're not the first ones to have issues with osslsigncode +
> > > nested signatures. https://github.com/mumble-voip/mumble/issues/1308 talks
> > > about it, and even points at code that is claimed to work. Unfortunately, I
> > > couldn't reproduce their results with our binaries. It's possible that their
> > > patchset only works on MSIs...
> > 
> > I looked into this some more, including speaking with the one of their
> > RelEng people, and it's looking like only Windows 8 and higher support
> > multiple signatures. When I look at a binary I've signed from Windows 8, I
> > see both signatures. AFAICT, Windows 7 and earlier only see the first
> > signature. 
> 
> Win7 with https://technet.microsoft.com/en-us/library/security/3033929
> should support dual signatures too.

It definitely supports SHA2, like that article says, but as far as I can tell it doesn't support multiple signatures on the same file. I've got some test files signed in http://people.mozilla.org/~bhearsum/sha1-tests.zip. If you can see both signatures on any of the "signed.exe" files in the "vanilla" directory on Windows 7, I'd be very interested. I can only see one on the Windows 7 machines I've tried on.
Works fine on Win7 for me with update installed.
(In reply to Yuhong Bao from comment #27)
> Created attachment 8635661 [details]
> dual signing on Win7 with update installed
> 
> Works fine on Win7 for me with update installed.

Thanks for verifying that!
(In reply to Yuhong Bao from comment #27)
> Created attachment 8635661 [details]
> dual signing on Win7 with update installed
> 
> Works fine on Win7 for me with update installed.

Hm, your statement just clicked with me. So it seems that some versions of Windows 7 will see both signatures. My own test was done with SP1. Unfortunately, we need to support early versions of Windows 7 too, so even though dual signatures works on more recent versions doesn't help here, I think :(.
(In reply to Ben Hearsum [:bhearsum] from comment #29)
> (In reply to Yuhong Bao from comment #27)
> > Created attachment 8635661 [details]
> > dual signing on Win7 with update installed
> > 
> > Works fine on Win7 for me with update installed.
> 
> Hm, your statement just clicked with me. So it seems that some versions of
> Windows 7 will see both signatures. My own test was done with SP1.
> Unfortunately, we need to support early versions of Windows 7 too, so even
> though dual signatures works on more recent versions doesn't help here, I
> think :(.
That is why I mentioned the update https://technet.microsoft.com/en-us/library/security/3033929
(In reply to Yuhong Bao from comment #30)
> (In reply to Ben Hearsum [:bhearsum] from comment #29)
> > (In reply to Yuhong Bao from comment #27)
> > > Created attachment 8635661 [details]
> > > dual signing on Win7 with update installed
> > > 
> > > Works fine on Win7 for me with update installed.
> > 
> > Hm, your statement just clicked with me. So it seems that some versions of
> > Windows 7 will see both signatures. My own test was done with SP1.
> > Unfortunately, we need to support early versions of Windows 7 too, so even
> > though dual signatures works on more recent versions doesn't help here, I
> > think :(.
> That is why I mentioned the update
> https://technet.microsoft.com/en-us/library/security/3033929

Relaying something else that Yuhong brought up on IRC today: it may be the case that users without this hotfix will continue to see sha1 fixes as valid. If that's true, multiple signatures (sha1 first, then sha2) may be an option. I'm trying to find a way to verify this.

Yuhong gave me a Microsoft contact, so I'm working with them at this point as well.
Depends on: 1186425
I've finally got a full Firefox build signed with our new SHA-2 cert: http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2015-07-23-07-32-38-oak/firefox-42.0a1.en-US.win32.installer.exe

One strange thing about it is that it doesn't run on any of my Windows installs with a double click (even the ones that recognize the signature), but it runs fine if I launch it through cmd.exe. Once installed, firefox.exe runs fine with a double click. I can't find anything in the Event Viewer about it, it seems very strange though.
(In reply to Ben Hearsum [:bhearsum] from comment #32)
> I've finally got a full Firefox build signed with our new SHA-2 cert:
> http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2015-07-23-07-32-38-
> oak/firefox-42.0a1.en-US.win32.installer.exe
> 
> One strange thing about it is that it doesn't run on any of my Windows
> installs with a double click (even the ones that recognize the signature),
> but it runs fine if I launch it through cmd.exe. Once installed, firefox.exe
> runs fine with a double click. I can't find anything in the Event Viewer
> about it, it seems very strange though.

I checked this file on my Win7 VM and it worked fine.
So I'm struggling a bit to formulate a recommendation. After talking with a bunch of people, my basic line of thinking is that anything that doesn't support SHA-2 is old enough and has few enough users that we should just live with a degraded experience on those platforms. However, I need to verify exactly what that experience will be before I can confidently make that recommendation.

As I understand it, the versions that don't support SHA-2 are unpatched versions of XP, Vista, and Windows 7.

The things I need to verify are:
* Initial install experience
* First run experience
* Update experience

What I'm mainly looking for is extra warnings or click-through dialogs that show up. I'm a bit worried that the behaviour I mentioned in comment #32 is a result of signing with a SHA-2 certificate.
(In reply to Ben Hearsum [:bhearsum] from comment #32)
> I've finally got a full Firefox build signed with our new SHA-2 cert:
> http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2015-07-23-07-32-38-
> oak/firefox-42.0a1.en-US.win32.installer.exe
> 
> One strange thing about it is that it doesn't run on any of my Windows
> installs with a double click (even the ones that recognize the signature),
> but it runs fine if I launch it through cmd.exe. Once installed, firefox.exe
> runs fine with a double click. I can't find anything in the Event Viewer
> about it, it seems very strange though.

That is exactly the Vista bug I was talking about. Which other versions of Windows does it show up?
(In reply to Yuhong Bao from comment #35)
> (In reply to Ben Hearsum [:bhearsum] from comment #32)
> > I've finally got a full Firefox build signed with our new SHA-2 cert:
> > http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2015-07-23-07-32-38-
> > oak/firefox-42.0a1.en-US.win32.installer.exe
> > 
> > One strange thing about it is that it doesn't run on any of my Windows
> > installs with a double click (even the ones that recognize the signature),
> > but it runs fine if I launch it through cmd.exe. Once installed, firefox.exe
> > runs fine with a double click. I can't find anything in the Event Viewer
> > about it, it seems very strange though.
> 
> That is exactly the Vista bug I was talking about. Which other versions of
> Windows does it show up?

This was on XP SP2.
(In reply to Ben Hearsum [:bhearsum] from comment #36)
> (In reply to Yuhong Bao from comment #35)
> > (In reply to Ben Hearsum [:bhearsum] from comment #32)
> > > I've finally got a full Firefox build signed with our new SHA-2 cert:
> > > http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2015-07-23-07-32-38-
> > > oak/firefox-42.0a1.en-US.win32.installer.exe
> > > 
> > > One strange thing about it is that it doesn't run on any of my Windows
> > > installs with a double click (even the ones that recognize the signature),
> > > but it runs fine if I launch it through cmd.exe. Once installed, firefox.exe
> > > runs fine with a double click. I can't find anything in the Event Viewer
> > > about it, it seems very strange though.
> > 
> > That is exactly the Vista bug I was talking about. Which other versions of
> > Windows does it show up?
> 
> This was on XP SP2.

Yea, I bet that most people still on it are existing users not new downloads, which would also allow the move to SHA2 for mozilla.org.
(In reply to Ben Hearsum [:bhearsum] from comment #34)
> As I understand it, the versions that don't support SHA-2 are unpatched
> versions of XP, Vista, and Windows 7.

Even Win7 RTM should be fine I think.
(In reply to Ben Hearsum [:bhearsum] from comment #36)
> (In reply to Yuhong Bao from comment #35)
> > (In reply to Ben Hearsum [:bhearsum] from comment #32)
> > > I've finally got a full Firefox build signed with our new SHA-2 cert:
> > > http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2015-07-23-07-32-38-
> > > oak/firefox-42.0a1.en-US.win32.installer.exe
> > > 
> > > One strange thing about it is that it doesn't run on any of my Windows
> > > installs with a double click (even the ones that recognize the signature),
> > > but it runs fine if I launch it through cmd.exe. Once installed, firefox.exe
> > > runs fine with a double click. I can't find anything in the Event Viewer
> > > about it, it seems very strange though.
> > 
> > That is exactly the Vista bug I was talking about. Which other versions of
> > Windows does it show up?
> 
> This was on XP SP2.

In both cases right clicking and clicking Unblock should work, yes?
(In reply to Yuhong Bao from comment #39)
> (In reply to Ben Hearsum [:bhearsum] from comment #36)
> > (In reply to Yuhong Bao from comment #35)
> > > (In reply to Ben Hearsum [:bhearsum] from comment #32)
> > > > I've finally got a full Firefox build signed with our new SHA-2 cert:
> > > > http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2015-07-23-07-32-38-
> > > > oak/firefox-42.0a1.en-US.win32.installer.exe
> > > > 
> > > > One strange thing about it is that it doesn't run on any of my Windows
> > > > installs with a double click (even the ones that recognize the signature),
> > > > but it runs fine if I launch it through cmd.exe. Once installed, firefox.exe
> > > > runs fine with a double click. I can't find anything in the Event Viewer
> > > > about it, it seems very strange though.
> > > 
> > > That is exactly the Vista bug I was talking about. Which other versions of
> > > Windows does it show up?
> > 
> > This was on XP SP2.
> 
> In both cases right clicking and clicking Unblock should work, yes?

There is no unblock button on XP SP2. I don't think anything is being blocked though, I'm pretty sure this is https://support.microsoft.com/en-us/kb/2763674, which talks about the function that extracts the certificate being unable to cope with SHA256 certificates.
(In reply to Ben Hearsum [:bhearsum] from comment #40)
> (In reply to Yuhong Bao from comment #39)
> > (In reply to Ben Hearsum [:bhearsum] from comment #36)
> > > (In reply to Yuhong Bao from comment #35)
> > > > (In reply to Ben Hearsum [:bhearsum] from comment #32)
> > > > > I've finally got a full Firefox build signed with our new SHA-2 cert:
> > > > > http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2015-07-23-07-32-38-
> > > > > oak/firefox-42.0a1.en-US.win32.installer.exe
> > > > > 
> > > > > One strange thing about it is that it doesn't run on any of my Windows
> > > > > installs with a double click (even the ones that recognize the signature),
> > > > > but it runs fine if I launch it through cmd.exe. Once installed, firefox.exe
> > > > > runs fine with a double click. I can't find anything in the Event Viewer
> > > > > about it, it seems very strange though.
> > > > 
> > > > That is exactly the Vista bug I was talking about. Which other versions of
> > > > Windows does it show up?
> > > 
> > > This was on XP SP2.
> > 
> > In both cases right clicking and clicking Unblock should work, yes?
> 
> There is no unblock button on XP SP2. 
There should be if you downloaded it using a browser is aware of XP SP2's new features.
Depends on: 1193310
Quick update here: we're still waiting on a decision to be made about whether or not we'll continue XP SP2 support.

I've done some additional testing, and AFAICT we'll only need a separate installer to support XP SP2. Once installed, Firefox runs fine even if it's only signed by a SHA-2 certificate. Updates also work without issue.
Still no decision here. 

If anyone else wants to do some testing, I've published an installer that's SHA-1 signed which contains a SHA-2 signed Firefox, and will update to a newer SHA-2 signed build. My testing was done as follows:
On XP SP2 (64-bit) machine:
1) Downloaded http://people.mozilla.org/~bhearsum/sha1/firefox-43.0a1.en-US.win32.installer.sha1.exe
2) Checked signature, looked fine: http://people.mozilla.org/~bhearsum/sattap/25ff42e9.png
3) Ran installer, got the usual "Security Warning" dialog: http://people.mozilla.org/~bhearsum/sattap/e9b01e46.png
4) Clicked through installer, taking all of defaults except for launching Firefox immediately
5) Browsed to installation location and checked signature of firefox.exe, which was invalid because of sha2: http://people.mozilla.org/~bhearsum/sattap/220f7cec.png
6) Launched Firefox with Desktop icon. No extra warnings or dialogs were shown.
7) Set app.update.log to true, opened Browser Console, and checked for updates
8) Update was successfully found and applied in the background: http://people.mozilla.org/~bhearsum/sattap/62083312.png
9) Restarted for update without issue
Attached patch switch mozilla-central to sha2 (obsolete) — — Splinter Review 
Still no decision about we need to for release, but I'm going to go ahead and switch Nightly to SHA-2 only next week, so it can start riding the trains. We need to make a decision about release before Firefox 43 hits Beta on November 11th, and hopefully well before that.

This patch will do that.
Attachment #8657300 - Flags: review?(catlee)
Attachment #8657300 - Flags: review?(catlee) → review+
Attachment #8657300 - Flags: checked-in+
Backed out in https://hg.mozilla.org/integration/mozilla-inbound/rev/e242dda37821 for windows xpcshell failures like https://treeherder.mozilla.org/logviewer.html#?job_id=13791146&repo=mozilla-inbound

On this bug's landing, that first set of tests was green, but it was orange on the following push, and stayed orange even after that following push was backed out.
Flags: needinfo?(bhearsum)
(In reply to Wes Kocher (:KWierso) from comment #46)
> Backed out in
> https://hg.mozilla.org/integration/mozilla-inbound/rev/e242dda37821 for
> windows xpcshell failures like
> https://treeherder.mozilla.org/logviewer.html#?job_id=13791146&repo=mozilla-
> inbound
> 
> On this bug's landing, that first set of tests was green, but it was orange
> on the following push, and stayed orange even after that following push was
> backed out.

Sorry about that. Looks like it's failing an "is binary signed" check, eg:
11:44:44  WARNING -  TEST-UNEXPECTED-FAIL | toolkit/mozapps/update/tests/unit_service_updater/marFileLockedStageFailureCompleteSvc_win.js | shouldRunServiceTest - [shouldRunServiceTest : 1745] the updater.exe binary should be signed (if not, build system configuration bug?) - false == true

I traced this down a bit, and it looks like that check ends up being done by https://dxr.mozilla.org/mozilla-central/source/toolkit/mozapps/update/tests/data/xpcshellUtilsAUS.js#1762, which running a hepler binary with "check-signature" as the first arg, which ends up at https://dxr.mozilla.org/mozilla-central/source/toolkit/mozapps/update/common/certificatecheck.cpp#211. I don't really understand that code, but I _think_ it's verifying the signature based on the certificates and policies on the test machine. Brian, can you confirm that?

If that's the case, we may need a hotfix on our test machines to allow them to validate SHA-2 signatures.
Flags: needinfo?(bhearsum) → needinfo?(netzen)
Looping in rstrong.
Note that this is about binary signing changes and not mar signing changes.

> I don't really understand that code, but I _think_ it's verifying the signature based on the certificates and policies on the test machine. Brian, can you confirm that?

I think that's probably the case.
Flags: needinfo?(netzen) → needinfo?(robert.strong.bugs)
(In reply to Brian R. Bondy [:bbondy] from comment #48)
> Looping in rstrong.
> Note that this is about binary signing changes and not mar signing changes.
Thanks!

> 
> > I don't really understand that code, but I _think_ it's verifying the signature based on the certificates and policies on the test machine. Brian, can you confirm that?
> 
> I think that's probably the case.
Yes.

We also have a certificate check before launching the post update process when using the service. Not sure what the best path for this is but probably either disabling the check on the affected platforms or not using the service for those platforms. :(
Flags: needinfo?(robert.strong.bugs)
(In reply to Robert Strong [:rstrong] (use needinfo to contact me) from comment #49)
> (In reply to Brian R. Bondy [:bbondy] from comment #48)
> > Looping in rstrong.
> > Note that this is about binary signing changes and not mar signing changes.
> Thanks!
> 
> > 
> > > I don't really understand that code, but I _think_ it's verifying the signature based on the certificates and policies on the test machine. Brian, can you confirm that?
> > 
> > I think that's probably the case.
> Yes.
> 
> We also have a certificate check before launching the post update process
> when using the service. Not sure what the best path for this is but probably
> either disabling the check on the affected platforms or not using the
> service for those platforms. :(

Disabling the check seems less than ideal...if we got the right hotfix(es) installed, we could leave it enabled, right? If so, I'll look into what it would take for that to happen.
Flags: needinfo?(robert.strong.bugs)
I'm talking about the clients since the check will fail and the post update process won't run
Flags: needinfo?(robert.strong.bugs)
(In reply to Wes Kocher (:KWierso) from comment #46)
> Backed out in
> https://hg.mozilla.org/integration/mozilla-inbound/rev/e242dda37821 for
> windows xpcshell failures like
> https://treeherder.mozilla.org/logviewer.html#?job_id=13791146&repo=mozilla-
> inbound
> 
> On this bug's landing, that first set of tests was green, but it was orange
> on the following push, and stayed orange even after that following push was
> backed out.

So, windows xpcshell tests on the backout were still orange. But I retriggered the Windows build, and the xpcshell test that got triggered from that new build came back green. 

Not sure what that says about our build system...
(In reply to Wes Kocher (:KWierso) from comment #52)
> (In reply to Wes Kocher (:KWierso) from comment #46)
> > Backed out in
> > https://hg.mozilla.org/integration/mozilla-inbound/rev/e242dda37821 for
> > windows xpcshell failures like
> > https://treeherder.mozilla.org/logviewer.html#?job_id=13791146&repo=mozilla-
> > inbound
> > 
> > On this bug's landing, that first set of tests was green, but it was orange
> > on the following push, and stayed orange even after that following push was
> > backed out.
> 
> So, windows xpcshell tests on the backout were still orange. But I
> retriggered the Windows build, and the xpcshell test that got triggered from
> that new build came back green. 
> 
> Not sure what that says about our build system...

That's so weird...I checked the most recent log and it was definitely making the right calls to the signing server (using osslsigncode, which should get it back SHA-1 signed binaries). I see that things cleared up on their own though...so maybe there'
(In reply to Wes Kocher (:KWierso) from comment #52)
> (In reply to Wes Kocher (:KWierso) from comment #46)
> > Backed out in
> > https://hg.mozilla.org/integration/mozilla-inbound/rev/e242dda37821 for
> > windows xpcshell failures like
> > https://treeherder.mozilla.org/logviewer.html#?job_id=13791146&repo=mozilla-
> > inbound
> > 
> > On this bug's landing, that first set of tests was green, but it was orange
> > on the following push, and stayed orange even after that following push was
> > backed out.
> 
> So, windows xpcshell tests on the backout were still orange. But I
> retriggered the Windows build, and the xpcshell test that got triggered from
> that new build came back green. 
> 
> Not sure what that says about our build system...

That's so weird...I checked the most recent log and it was definitely making the right calls to the signing server (using osslsigncode, which should get it back SHA-1 signed binaries). I see that things cleared up on their own though...so maybe there's some busted caching somewhere.
(In reply to Robert Strong [:rstrong] (use needinfo to contact me) from comment #51)
> I'm talking about the clients since the check will fail and the post update
> process won't run

This is interesting. I tested updates from a SHA-1 -> SHA-2 signed build on XP SP2 (which has no SHA-2 support) explicitly, but I didn't test SHA-2 -> SHA-2 anywhere. Are you saying that updates simply will not work on certain platforms if the updater binary is incorrectly signed?
Flags: needinfo?(robert.strong.bugs)
Something Rail pointed out to me is that we probably need to install the new self signed CA onto our Windows machines, too. Doing so would fix the test failures of dep/ci builds, but probably not for nightlies or releases.

Even so, we have to figure out what implications there are for real world users that don't have the DigiCert SHA-2 root on their machine.
(In reply to Ben Hearsum (:bhearsum) from comment #55)
> (In reply to Robert Strong [:rstrong] (use needinfo to contact me) from
> comment #51)
> > I'm talking about the clients since the check will fail and the post update
> > process won't run
> 
> This is interesting. I tested updates from a SHA-1 -> SHA-2 signed build on
> XP SP2 (which has no SHA-2 support) explicitly, but I didn't test SHA-2 ->
> SHA-2 anywhere. Are you saying that updates simply will not work on certain
> platforms if the updater binary is incorrectly signed?
I am fairly certain that the cert check will fail and the cert check passing is required to use the service and to run the post update process with the service. There are tons of fallbacks to successfully complete the update and you likely successfully updated using a fallback that didn't use the service.
Flags: needinfo?(robert.strong.bugs)
Attachment #8657300 - Flags: checked-in+ → checked-in-
I spoke with Robert today just now to try and figure out to verify all of the things he's concerned about. A few things came up:
* Updates through the Maintenance Service will not work if all the following are true: the user is not an admin, the service is installed + enabled, the signature on the updater cannot be verified. The latter will be true for all XP SP2 users, so that's the main group we need to be concerned with here. AFAICT, even Windows 7 without any service packs applied should be able to validate the signature on the updater.
* I need to regenerate our self signed sha2 root+cert, because the issuer details on the cert are different than the old one, and Firefox requires certain values to be in place there. This should be quick and easy. Once this is done, running a try build w/ sha2 signing enabled, and running xpcshell + mochitest-other will run all of the relevant unit tests around the updater and maintenance service
* He also suggested doing a build with sha2 enabled on Oak, which will use our DigiCert SHA-2 cert and run the necessary unit tests.
* I'm going to set up some manual update paths that let us install a SHA-1 signed build, update to a SHA-2 signed build, and then update to another SHA-2 signed build. This should let me and others test a variety of update scenarios. Robert suggested the following procedure for manual tests:
1) Disable the Background service in Firefox prefs
2) Attempt to update, verify that a UAC prompt shows up, don't allow the update to proceed.
3) Re-enable the Background service
4) Attempt to update again, verify that the update happens without a UAC prompt, and that the browser actually updates.
(In reply to Ben Hearsum (:bhearsum) from comment #58)
> I spoke with Robert today just now to try and figure out to verify all of
> the things he's concerned about. A few things came up:
> * I need to regenerate our self signed sha2 root+cert, because the issuer
> details on the cert are different than the old one, and Firefox requires
> certain values to be in place there. This should be quick and easy. Once
> this is done, running a try build w/ sha2 signing enabled, and running
> xpcshell + mochitest-other will run all of the relevant unit tests around
> the updater and maintenance service

I've regenerated the self signed cert. Tomorrow I'll get it deployed and do a try push with it.

> * He also suggested doing a build with sha2 enabled on Oak, which will use
> our DigiCert SHA-2 cert and run the necessary unit tests.

I pushed this as https://treeherder.mozilla.org/#/jobs?repo=oak&revision=a7e06f6ac5ba
Depends on: 1203990
Depends on: 1204081
OK, I've got a new test set-up that starts from a SHA-1 external+internal -> SHA-2 internal -> SHA-2 internal. I did two tests on Windows XP SP2, one with the background service and enabled and one with it disabled. Both tests were performed as Administrator. I followed the instructions from comment #58, but UAC doesn't apply to XP SP2, so I based pass/fail on update log messages rather than UAC dialogs. In both cases, XP SP2 was able to fully update, despite the updater binary not having a valid signature for the second update. I'm guessing that this is because while the _signature_ is unrecognized, the certificate it was made with can be traced back Windows' built-in DigiCert root. I'd really appreciate more eyes on these, particularly if someone has access to a Windows 7 machine that doesn't support SHA-2. My version came with SP1, and sees the SHA-2 binaries as validly signed.

Here's the gory details of the tests, with screenshots:
XP SP2 w/ Background Service:
* Install http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2015/08/2015-08-14-04-02-05-oak/firefox-42.0a1.en-US.win32.installer.exe
** Observe that it is validly signed: http://people.mozilla.org/~bhearsum/sattap/d2a7ac31.png, http://people.mozilla.org/~bhearsum/sattap/9563e708.png
* Launch Firefox, immediately set app.update.log to true and open Browser Console
** Catch the tail end of update messages, showing that that the service is installed and able to stage updates. Also note the "updated" folder in the appdir: http://people.mozilla.org/~bhearsum/sattap/7a8858e2.png
* Restart Firefox and open the Browser Console immediately. Also check signature of updater.exe.
** Catch tail end of the next update request, see it succeed with the background service: http://people.mozilla.org/~bhearsum/sattap/1b689597.png, http://people.mozilla.org/~bhearsum/sattap/e5669f90.png
* Restart Firefox and open the Browser Console immediately. Also check signature of updater.exe.
** No updates available: http://people.mozilla.org/~bhearsum/sattap/c21f4eae.png


XP SP2 w/out Background Service:
* Install http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2015/08/2015-08-14-04-02-05-oak/firefox-42.0a1.en-US.win32.installer.exe
** Observe that it is validly signed: http://people.mozilla.org/~bhearsum/sattap/bc28a327.png
* Launch Firefox, set app.update.log to true, and disable the Background Service
* Close Firefox and get rid of half finished background update
** Delete "updated" from appdir; "Nightly" from ~Local Settings\Application Data\Mozilla\Firefox
* Start Firefox and open the Browser Console immediately. Also check the signature of updater.exe.
** Catch the tail end of the update request, see it succeed _without_ the background service: http://people.mozilla.org/~bhearsum/sattap/ce61a8f5.png
* Restart for the update and open the Browsole Console again.
** Catch the next update request, see it succeed _without_ the background service and with an updater.exe who's signature cannot be validated: http://people.mozilla.org/~bhearsum/sattap/e93a27e2.png, http://people.mozilla.org/~bhearsum/sattap/ef1259d4.png
* Restart again -> No updates available.
(In reply to Ben Hearsum (:bhearsum) from comment #59)
> (In reply to Ben Hearsum (:bhearsum) from comment #58)
> > * He also suggested doing a build with sha2 enabled on Oak, which will use
> > our DigiCert SHA-2 cert and run the necessary unit tests.
> 
> I pushed this as
> https://treeherder.mozilla.org/#/jobs?repo=oak&revision=a7e06f6ac5ba

I finally got results on this. Something interesting here is that the test failures are different between the self signed builds and the ones signed by the DigiCert cert. Specifically, the self signed builds fail 30 XPCShell tests, and the DigiCert one only fails 1:
10:47:23  WARNING -  TEST-UNEXPECTED-FAIL | toolkit/mozapps/update/tests/unit_service_updater/checkUpdaterSigSvc.js | xpcshell return code: 0
10:47:23     INFO -  TEST-INFO took 250ms
10:47:23     INFO -  >>>>>>>
10:47:23     INFO -  (xpcshell/head.js) | test MAIN run_test pending (1)
10:47:23  WARNING -  TEST-UNEXPECTED-FAIL | toolkit/mozapps/update/tests/unit_service_updater/checkUpdaterSigSvc.js | run_test - [run_test : 39] the maintenance service exit value should be 0 - 1 == 0
10:47:23     INFO -  C:/slave/test/build/tests/xpcshell/tests/toolkit/mozapps/update/tests/unit_service_updater/checkUpdaterSigSvc.js:run_test:39
10:47:23     INFO -  C:\slave\test\build\tests\xpcshell\head.js:_execute_test:510
10:47:23     INFO -  -e:null:1
10:47:23     INFO -  null:null:0
10:47:23     INFO -  exiting test
10:47:23     INFO -  <<<<<<<


I'm not terribly surprised that the DigiCert one doesn't fail as many tests -- its cert is issued by a cert that is ultimately trusted by Windows, whereas the self signed sha2 isn't (until bug 1203990 is fixed). I'm actually a little surprised this test fails, because files signed by the DigiCert SHA-2 cert look fine to me when viewing the signatures in Explorer on our test machines.
(In reply to Ben Hearsum (:bhearsum) from comment #61)
> I'm not terribly surprised that the DigiCert one doesn't fail as many tests
> -- its cert is issued by a cert that is ultimately trusted by Windows,
> whereas the self signed sha2 isn't (until bug 1203990 is fixed). I'm
> actually a little surprised this test fails, because files signed by the
> DigiCert SHA-2 cert look fine to me when viewing the signatures in Explorer
> on our test machines.

Looking at https://dxr.mozilla.org/mozilla-central/source/toolkit/mozapps/update/common/registrycertificates.cpp?offset=1100#28 (which is what the test calls into) it reads like it looks up the maintenance service from the registry, which makes me wonder if the test is looking at the wrong file? I'm probably wrong about that, just a thought though...
Robert - could you have a quick look at comment #60 and see what you think of my tests? I'm going to get Softvision to do a bunch of additional tests on OS' I don't have access to, so I want to make sure the instructions are good first. Obviously things are a little bit different for OS' w/ UAC, though.
Flags: needinfo?(robert.strong.bugs)
Aha, I was finally able to make an update fail on Windows 7 (no service packs). I followed the same instructions as on XP, but it failed when trying to stage the update (the first one to be done with the sha-2 signed updater.exe) with error 26 (which is SERVICE_UPDATER_SIGN_ERROR, according to https://github.com/mozilla/gecko-dev/blob/master/toolkit/mozapps/update/common/errors.h

After digging into this more, I see this is because there's registry entries that specify the issuer of the certificate that signed the Maintenance Service (or maybe updater.exe) binary. So it seems like we need to make sure that we update that at the same time as switching to a SHA-2 certificate. It looks like they're defined in the maintenance service NSI (https://dxr.mozilla.org/mozilla-central/source/toolkit/components/maintenanceservice/bootstrapinstaller/maintenanceservice_installer.nsi?offset=0#220), so I'm not sure how we'd make changes to them as part of our update process...
Depends on: 1205843
Removing needinfo from Rob because I sorted out test instructions myself -- Softvision has almost finished running them. So far, nothing new has come up (just confirmation of bug 1205843). 

Request tracking for 43 because we need to upgrade to SHA-2 before the end of the year.
tracking-firefox43: --- → ?
Flags: needinfo?(robert.strong.bugs)
Attached patch updated patch to enable sha2 (obsolete) — — Splinter Review 
Here's an updated patch for Gecko. This will enable SHA-2 signing and update all of the issuer hard codes we have for stub installer + maintenance service - so new installs will work 100% correctly (including background updates) with this patch. It's not ready for review because we may need additional changes depending on what happens with bug 1205843 to support existing installs.
Attachment #8657300 - Attachment is obsolete: true
bug 1203990 got the new self signed root deployed to our test machines recently. My latest push to oak is passing tests on the CI builds: https://treeherder.mozilla.org/#/jobs?repo=oak&revision=2e0e6bcef472, so that's good. Still have to deal with bug 1205843 before we can switch over.
I'm going to be off for awhile, so I'm handing this off to Rail. I've fully briefed him on all of the background.
Assignee: bhearsum → rail
We'll need something like this for partner repacks, too. Can't land until we switch Firefox over to SHA-2 and it gets uplifted to Beta.
Based on what I understand this will really come into effect 44 on wards, so 44+ makes sense.
status-firefox44: --- → affected
tracking-firefox44: --- → +
(In reply to Ritu Kothari (:ritu) from comment #70)
> Based on what I understand this will really come into effect 44 on wards, so
> 44+ makes sense.

43, actually. Because 44 doesn't ship until after January 1st 2016, we must be able to sign an potential 43 point releases with SHA-2, which means we should ship 43.0 with SHA-2.
Flags: needinfo?(rkothari)
Depends on: 1211317
Depends on: 1226191
Hi catlee, you suggested landing https://bug1079858.bmoattachments.org/attachment.cgi?id=8666874 today so that we can test this in Nightly. Can you take care of that or should someone else go ahead?
Flags: needinfo?(catlee)
Flags: needinfo?(catlee)
Attachment #8666874 - Flags: review+
Attachment #8666874 - Flags: checked-in+
Matt, Florin, just a heads up that this should land in tomorrow's nightly.  I think we should test the Windows update process and verify that it's working in nightly, then aurora, then uplift any more changes to beta and test there.    Can you test and let us know the results?

We don't have a lot of time left before the workweek and the release, but I'm hoping that we will be very sure Windows updates will work with SHA-1 for 43 release.
Flags: needinfo?(mhowell)
Flags: needinfo?(florin.mezei)
The timing seems to be really tight here. Adding Andrei to see what we can do here (either tomorrow or on Monday).
Flags: needinfo?(florin.mezei) → needinfo?(andrei.vaida)
https://hg.mozilla.org/mozilla-central/rev/0ea0832d052d
Status: NEW → RESOLVED
Closed: 9 years ago
status-firefox45: --- → fixed
Resolution: --- → FIXED
fyi - win32 nightlies against patch: https://bugzilla.mozilla.org/show_bug.cgi?id=1079858#c74 are finished 

builds and artifacts can be found: http://archive.mozilla.org/pub/firefox/nightly/latest-mozilla-central/

note: win64 are still running so the current win64 builds in the above location are old.
> note: win64 are still running so the current win64 builds in the above
> location are old.

win 64 builds are now up to date and beside win32 in: http://archive.mozilla.org/pub/firefox/nightly/latest-mozilla-central/
I've run updates to the new nightlies from older nightly builds on Windows 10 (64-bit) and Windows XP SP2 (32-bit), and both updates succeeded normally (that is, with no unusual prompts or anything like that). I'll fill in that range with other Windows versions shortly and report back here, but preliminarily things look good.
(In reply to Matt Howell [:mhowell] from comment #80)
> I've run updates to the new nightlies from older nightly builds on Windows
> 10 (64-bit) and Windows XP SP2 (32-bit), and both updates succeeded normally
> (that is, with no unusual prompts or anything like that). I'll fill in that
> range with other Windows versions shortly and report back here, but
> preliminarily things look good.

thanks matt.

plan of uplift action:

liz and I discussed this over vidyo and we are not going to uplift this to beta for beta9. Instead, we will uplift this to just aurora today and test aurora nightly tomorrow.

prior to friday's uplift m-b->m-r uplift, we can uplift this to beta. That way on monday, for the RC beta release, we can test this works on the beta channel.
(In reply to Matt Howell [:mhowell] from comment #80)
> I've run updates to the new nightlies from older nightly builds on Windows
> 10 (64-bit) and Windows XP SP2 (32-bit), and both updates succeeded normally
> (that is, with no unusual prompts or anything like that). I'll fill in that
> range with other Windows versions shortly and report back here, but
> preliminarily things look good.

All right, I've now run updates from the previous to the current nightly on all of the following Windows versions, and all of them succeeded:

XP SP2 (32-bit build)
XP SP3 (32-bit build)
Vista unpatched (32-bit build)
Vista with current updates (32-bit build)
7 unpatched (32-bit build)
7 with current updates (32-bit build)
8.1 (32-bit build)
10 (32-bit and 64-bit builds)

That's every version that I have access to, so I've done all the testing I can.
Flags: needinfo?(mhowell)
It would be a good thing to also update to the same version from the version with the changes. This can be done by copying the update.xml to your own server, removing the partial patch entry, and setting the app.update.url.override pref to point at the custom update.xml.
(In reply to Robert Strong [:rstrong] (use needinfo to contact me) from comment #83)

Might need to increase the buildID in the xml too, per
 http://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/update/nsUpdateService.js#2818
(In reply to Robert Strong [:rstrong] (use needinfo to contact me) from comment #83)
> It would be a good thing to also update to the same version from the version
> with the changes. This can be done by copying the update.xml to your own
> server, removing the partial patch entry, and setting the
> app.update.url.override pref to point at the custom update.xml.

andrei, would you be able to help us test this and any other QA you think required?

We are going to try to uplift this to aurora now and beta tomorrow.
Blocks: 1206668
Comment on attachment 8666874 [details] [diff] [review]
updated patch to enable sha2

Approval Request Comment
[Feature/regressing bug #]: Bug 1226191
[User impact if declined]: this is required for windows xp users. will break install if not landed
[Describe test coverage new/current, TreeHerder]: https://bugzilla.mozilla.org/show_bug.cgi?id=1079858#c82
[Risks and why]: 
[String/UUID change made/needed]:
Attachment #8666874 - Flags: approval-mozilla-beta?
Attachment #8666874 - Flags: approval-mozilla-aurora?
Comment on attachment 8666874 [details] [diff] [review]
updated patch to enable sha2

Please uplift to aurora and beta.
Attachment #8666874 - Flags: approval-mozilla-beta?
Attachment #8666874 - Flags: approval-mozilla-beta+
Attachment #8666874 - Flags: approval-mozilla-aurora?
Attachment #8666874 - Flags: approval-mozilla-aurora+
Ben, Jordan, should we add this to 43 release notes?
Flags: needinfo?(jlund)
(In reply to Ritu Kothari (:ritu) from comment #89)
> Ben, Jordan, should we add this to 43 release notes?

seems like something we want to mention as SHA1 is deprecated. but there are likely bigger players involved in this who can weigh in. Ben is away until Mozlando
Flags: needinfo?(jlund)
We're going to have to back this patch out because of bug 1230631. This patch created a chicken-and-egg situation where our updater is checking certificates on SHA-2 signed files by looking for the old issuer information from the SHA-1 certificate.

What we'll need to do is break this patch into two parts, one that updates the certificate pinning information, and a second one that actually enables SHA-2. Those two patches then need to go out in subsequent releases to each channel, *not* all at the same time as the current patch does. I'll make those patches and add them here.
And trunk:
https://hg.mozilla.org/mozilla-central/rev/5ba77225c957

Hopefully Nightly builds are able to downgrade to the sha1 builds without problems. :)
Status: RESOLVED → REOPENED
status-firefox45: fixed → affected
Resolution: FIXED → ---
(In reply to Wes Kocher (:KWierso) from comment #94)
> And trunk:
> https://hg.mozilla.org/mozilla-central/rev/5ba77225c957
> 
> Hopefully Nightly builds are able to downgrade to the sha1 builds without
> problems. :)

should we lock nightly now to the nightly before this patch was applied so no users who haven't upgraded get it? and do some manual testing in the mean time with verifying we can downgrade back to sha1 ?
Assignee: rail → mhowell
Status: REOPENED → ASSIGNED
Attachment #8666874 - Attachment is obsolete: true
(In reply to Jordan Lund (:jlund) from comment #95)
> (In reply to Wes Kocher (:KWierso) from comment #94)
> > And trunk:
> > https://hg.mozilla.org/mozilla-central/rev/5ba77225c957
> > 
> > Hopefully Nightly builds are able to downgrade to the sha1 builds without
> > problems. :)
> 
> should we lock nightly now to the nightly before this patch was applied so
> no users who haven't upgraded get it? and do some manual testing in the mean
> time with verifying we can downgrade back to sha1 ?

I've locked windows only users to 20151203030226 on m-c nightly until we test the above ^
Is the requested testing in comment 86 still needed due to the backouts? 
If so, could you please provide us an update.xml file with the necessary changes? We're not sure what to delete/add in it in order to test this. 

OR should we focus for now on testing that the users are able to downgrade back to sha1 (as per comment 95)?
Flags: needinfo?(robert.strong.bugs)
Flags: needinfo?(jlund)
Flags: needinfo?(andrei.vaida)
The requested testing in comment #86 is no longer needed and testing downgrading would be a goof thing.
Flags: needinfo?(robert.strong.bugs)
Should patch 1  be landing not just on nightly but on beta/release?  I was holding up the RC(1) build for it.
Flags: needinfo?(mhowell)
OK. We aren't going to land either patch in beta.  We should focus instead on making sure the downgrade is OK on nightly for the people who got this update.  And, then getting people on nightly and aurora both patches in the correct order.   

We will need to do two dot releases for 43 after next week (and before Jan. 1).
Matt, could you please comment on the effect of patch 1 on the stub installer. I'm concerned that the changes to browser/branding should be in patch 2, otherwise we'll change the stub's expectation before we actually change the signing.
(In reply to Nick Thomas [:nthomas] from comment #103)
> Matt, could you please comment on the effect of patch 1 on the stub
> installer. I'm concerned that the changes to browser/branding should be in
> patch 2, otherwise we'll change the stub's expectation before we actually
> change the signing.

I didn't think of this, but you're exactly right; dividing the patches this way would confuse the stub installer and cause it to fail. I'll obsolete the first pair of patches and move the branding changes into patch 2; that will be the only change. I'll do that now, while I pray that we're at least running out of problems.
Flags: needinfo?(mhowell)
Attachment #8696101 - Attachment is obsolete: true
Attachment #8696102 - Attachment is obsolete: true
Comment on attachment 8696727 [details] [diff] [review]
Part 1: Switch the updater to start expecting SHA-2 signatures in subsequent updates

https://hg.mozilla.org/mozilla-central/rev/ba03ae4ac5df
Attachment #8696727 - Flags: checked-in+
Comment on attachment 8696728 [details] [diff] [review]
Part 2: Use SHA-2 to sign Windows binaries, and switch the stub installer to expect that

https://hg.mozilla.org/mozilla-central/rev/a8965ae93c5d
Attachment #8696728 - Flags: checked-in+
32 and 64bit windows nightlies have been triggered on rev ba03ae4ac5df, with buildid 20151208132555.
(In reply to Jordan Lund (:jlund) from comment #98)
> I've locked windows only users to 20151203030226 on m-c nightly until we
> test the above ^

Looks like we locked updates for win64 (via build targets of WINNT_x86_64-msvc and WINNT_x86_64-msvc-x64) but missed win32. I've converted it to block all windows by blocking OS version with 'Windows_NT' instead.
And forced win32+win64 nightlies on rev a8965ae93c5d, with buildid 20151208203040. (Delay for l10n to have a chance to run)
Cornel & SoftVision, could we have some urgent testing please; you too mhowell if you have your set of VMs with you.

NB: We still waiting for the 2nd build to finish, which will be when there is a green N on each row of
  https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=a8965ae93c5d&filter-searchStr=nightly%20WINNT
Dark grey means it's still building, but once it's green you can start testing. This should be at about 00:30 Pacific/10:10am Bucharest.


The testing we'd like done is
* install the win32 build from
   https://archive.mozilla.org/pub/firefox/nightly/2015/12/2015-12-08-03-02-12-mozilla-central/firefox-45.0a1.en-US.win32.installer.exe
* change the update channel to nightlytest in <install_dir>/defaults/pref/channel-prefs.js. You will probably need to modify the perissions so you can write to the file
* check for updates. You should get an update to ba03ae4ac5df, with buildid 20151208132555   [2]
* check for updates again. You should then get another update to a8965ae93c5d, with buildid 20151208203040

We would like to know if the second update causes any UAC prompts, or other UI asking for permission to continue, warning about untrusted applications etc etc. And of course any other problems with updating. It would be good to cover Windows 10, Win 8.1 or 8, Win 7, Vista, and XP SP2 or higher, depending on what you have available.

It would also be helpful to verify win64 using
 https://archive.mozilla.org/pub/firefox/nightly/2015/12/2015-12-08-03-02-12-mozilla-central/firefox-45.0a1.en-US.win64.installer.exe
where it's possible to (ie don't worry about 64bit XP).

Many thanks.


[1] this starting build is from rev
      https://hg.mozilla.org/mozilla-central/rev/2bdd9ec79799eff3ceec0a318f5a0632d918a527
    ie after the backout and should be equivalent to before we first landed on m-c
[2] RelEng - I added a new rule for nightlytest, find it by filtering on 'Firefox nightlytest'
[3] I did a quick check of the first update on Win10 Update 1, and it worked OK, with 
      HKLM\SOFTWARE\Mozilla\MaintenanceService\a4948786b022cc1a376991b44efb838c\0\issuer
    getting updated to 'DigiCert SHA2 Assured ID Code Signing CA'. So that's promising.
Flags: needinfo?(cornel.ionce)
(In reply to Nick Thomas [:nthomas] from comment #112)
> This should be at about 00:30 Pacific/10:10am Bucharest.

Oops, make that 00:30a Pacific/10:30am Bucharest.
I've managed to complete the requested testing.
In all cases, for each tested OS, the first update (to ba03ae4ac5df, with buildid 20151208132555) is skipped and it goes straight for the second one.
There were no UAC prompts/permission requests during the update.
My detailed testing and results are found in the etherpad[1]. I've also wrote down the certification for each tested OS.

Also, removing my needinfo to Jordan, since it's no longer required.

[1] https://public.etherpad-mozilla.org/p/sha1-deprecation-bug1079858

Please let me know if there's anything else you'd like me to cover.
Flags: needinfo?(jlund)
Flags: needinfo?(cornel.ionce)
Sorry, I made a mistake with the rules, and we've fixed that now. After the second update you'll get the latest nightly build, which is now 20151209030228.

However, I tried this and there's a problem. The first update worked correctly, but the second hit 'failed: 26' (in the browser console when trying to stage the update with the service). This means SERVICE_UPDATER_SIGN_ERROR, and very likely comes from the registry specifying the SHA2 cert, while the maintenance service is looking at the sha1-signed updater.exe.

If I click Restart in the UI then I get a standard UAC prompt, and the update works. This was not scary on Win10, but don't know if that will also be the case after Jan 1.

I'm going to test if putting both certs in the registry works.
You've got nightly-only xpcshell bustage with the maintenance service, https://treeherder.mozilla.org/logviewer.html#?job_id=2857284&repo=mozilla-central
And also on https://treeherder.mozilla.org/logviewer.html#?job_id=2861717&repo=mozilla-central, which is the second nightly with both patches (rev 40b58759c962). I don't know anything about that, except that we have a special cert on the test machines. That may be relevant to testing on dep builds though. Deferring to mhowell on this.
Good news - having both certificates in the registry allows you to apply a SHA2 update with a SHA1 updater.exe.

I basically did comment #112 on win32, but exported 
 HKEY_LOCAL_MACHINE\SOFTWARE\Mozilla\MaintenanceService\a4948786b022cc1a376991b44efb838c\0
before the first update. Then after the first update modify 0 -> 1 in the registry (this contains the SHA2 cert details), and merge in my earlier export.
FTR, after two updates the registry looks like this:

[HKEY_LOCAL_MACHINE\SOFTWARE\Mozilla\MaintenanceService\a4948786b022cc1a376991b44efb838c\0]
"name"="Mozilla Corporation"
"issuer"="DigiCert SHA2 Assured ID Code Signing CA"

[HKEY_LOCAL_MACHINE\SOFTWARE\Mozilla\MaintenanceService\a4948786b022cc1a376991b44efb838c\1]
"name"="Mozilla Corporation"
"issuer"="DigiCert SHA2 Assured ID Code Signing CA"

ie two copies of the same info.
Attachment #8696727 - Attachment is obsolete: true
Comment on attachment 8697027 [details] [diff] [review]
Part 1: Inform the updater and maintenance service of the new SHA2 certicicate issuer

Backed out previous two landings
https://hg.mozilla.org/mozilla-central/rev/33b00efca5d5

Landed this patch:
https://hg.mozilla.org/mozilla-central/rev/9aa2dae27ae5
Attachment #8697027 - Flags: checked-in+
Comment on attachment 8697027 [details] [diff] [review]
Part 1: Inform the updater and maintenance service of the new SHA2 certicicate issuer

rstrong, could you please review and sanity check this. The previous few comments hopefully capture the state here, but we need need an update to put both certs in the registry. On the next update we switch to SHA2 signing, but need the SHA1 cert for the maintenance service to accept the updater.exe which is on disk, and the SHA2 helper.exe after the update (and future updates).


Win32 & Win64 nightlies triggered on 9aa2dae27ae5, with buildid 20151209095500.
Attachment #8697027 - Flags: review?(robert.strong.bugs)
Comment on attachment 8696728 [details] [diff] [review]
Part 2: Use SHA-2 to sign Windows binaries, and switch the stub installer to expect that

Backed out at http://hg.mozilla.org/mozilla-central/rev/33b00efca5d5. We can land this again in a few hours and do another set of nightlies.
Attachment #8696728 - Flags: checked-in+ → checked-in-
Is 64bit Nightly on purpose still stuck to 12-03...? Says "up to date"..
Sorry for spam if not this bug...
Comment on attachment 8697027 [details] [diff] [review]
Part 1: Inform the updater and maintenance service of the new SHA2 certicicate issuer

Looks fine and is the same thing I discussed with mhowell last week.

BTW: why the extra indentation on comments?
Attachment #8697027 - Flags: review?(robert.strong.bugs) → review+
> Looks fine and is the same thing I discussed with mhowell last week.
> 
> BTW: why the extra indentation on comments?

Thanks for taking a look.

The indentation is just a style that I use to sort of simulate paragraph indentation, but reversed so it looks right in code. I usually remember not to do it because it doesn't match anyone else's style, but this time I didn't think about it; sorry.
No problem and go ahead and fix that later when you have a chance
(In reply to Antti Tervasmäki from comment #124)
> Is 64bit Nightly on purpose still stuck to 12-03...? Says "up to date"..
> Sorry for spam if not this bug...

win32 and win64 won't update past 20151203030226 until we sort this bug out. Hopefully not much longer.
Comment on attachment 8696728 [details] [diff] [review]
Part 2: Use SHA-2 to sign Windows binaries, and switch the stub installer to expect that

Relanded at https://hg.mozilla.org/mozilla-central/rev/b40ba117fa75

Nightlies running with buildid 20151209142514.
Attachment #8696728 - Flags: checked-in- → checked-in+
As someone who has been updating manually recently (since before I knew about this bug), is there anything I need to do for automatic updates to work correctly after all of this is resolved? Do I need to go back to the 20151203030226 build and use auto-update so the maintenance service accepts the certificate, or will it Just Work™ after I replace the files in %ProgramFiles%\Nightly with those from the latest archive?
Updating on 32bit Win10 WFM for 20151208030212 -> 20151209095500 (part1) -> 20151209142514 (part2). No UAC prompts or other errors seen. Diff against a fresh install is clean (ie all the files were correctly updated).

Cornel, could you please have another go at the request in comment #112.
Flags: needinfo?(cornel.ionce)
Nightly users could be in any of these states
* older than when we first landed the code        - sha1 binaries, sha1 in registry
* updated to SHA2 nightly (~20151204) and stopped - sha2 binaries, sha1 in registry, have been getting UAC prompts
* did above and continued to post-backout nightly - sha1 binaries, sha1 in registry

Since we can't know what the user has, but hopefully most people are sha1-sha1 at this point due to the locking to 20151208, I suggest we serve everyone an update to part1/9aa2dae27ae5/20151209095500, and then onwards to the latest sha2 build. When I tested the 2nd scenario there was one more UAC prompt getting to part1, then it was silent.

If there are no problems found by SoftVision, or other objections, then lets do that tomorrow.
Completed the requested testing with details in the etherpad: https://public.etherpad-mozilla.org/p/sha1-deprecation-bug1079858-p2.

This time both updates passed for every OS tested and there were no UAC prompts whilst updating (only a couple when Nightly was installed).
Flags: needinfo?(cornel.ionce)
Fantastic, thanks Cornel. 

I've re-enabled updates on Nightly, with path
   old  --->   20151209095500 (9aa2dae27ae5)  ---> latest
Some users will get a UAC prompt on the first update, but then should not see any more. Please let us know if that's not the case.

At some point in the future we'll remove that watershed at 20151209095500.
¡Hola Matt!

Got two updates today ending up at Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0 ID:20151210030212 CSet: b40ba117fa757861c9caa660ae989122718b494b on the second one I saw an UAC prompt.

Please let me know if there's anything worth collecting from the system.

¡Gracias!
Flags: needinfo?(mhowell)
Alex, thank you for the report. Unfortunately the best information would have been available only before the second update was applied. If you would be able to reply with the contents of the registry key
HKEY_LOCAL_MACHINE\SOFTWARE\Mozilla\MaintenanceService\[long hex string]\
then I can at least tell you whether you should have any further problems.
Flags: needinfo?(mhowell) → needinfo?(alex_mayorga)
¡Hola Matt!

It is no big deal really IMHO.

There are two of those "[long hex string]" is that normal?

Please find the contents of the keys FWIW.

¡Gracias!


Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Mozilla\MaintenanceService]
"Attempted"=dword:00000001
"Installed"=dword:00000001

[HKEY_LOCAL_MACHINE\SOFTWARE\Mozilla\MaintenanceService\b2ab7624dadf2b6829dcf48a806f2e0a]

[HKEY_LOCAL_MACHINE\SOFTWARE\Mozilla\MaintenanceService\b2ab7624dadf2b6829dcf48a806f2e0a\0]
"name"="Mozilla Corporation"
"issuer"="DigiCert SHA2 Assured ID Code Signing CA"

[HKEY_LOCAL_MACHINE\SOFTWARE\Mozilla\MaintenanceService\b2ab7624dadf2b6829dcf48a806f2e0a\1]
"name"="Mozilla Corporation"
"issuer"="DigiCert Assured ID Code Signing CA-1"

[HKEY_LOCAL_MACHINE\SOFTWARE\Mozilla\MaintenanceService\d13c27fed1934d7ca63c2f00af84c38a]

[HKEY_LOCAL_MACHINE\SOFTWARE\Mozilla\MaintenanceService\d13c27fed1934d7ca63c2f00af84c38a\0]
"name"="Mozilla Corporation"
"issuer"="DigiCert Assured ID Code Signing CA-1"
Flags: needinfo?(alex_mayorga) → needinfo?(mhowell)
(In reply to alex_mayorga from comment #138)

Okay, you seem to have been in some kind of interesting state regarding multiple installations (or possibly an incomplete uninstallation) prior to these updates, but you'll be okay moving forward; you shouldn't get any more of those prompts. Thanks again.
Flags: needinfo?(mhowell)
Depends on: 1231897
Nick, around breakfast, we realized that we haven't done anything for esr... :/
We probably should land a patch for this, right?
Flags: needinfo?(nthomas)
rstrong, can we use telemetry to look at how many times we don't use the service for Nightly updates, and general failure rates ? Particularly if we get buildid information.
Flags: needinfo?(robert.strong.bugs)
Sylvestre, we haven't landed beyond m-c at this point, and should generate a plan asap (see also request to rstrong above). For release it looks like we'll land patch1 for a 43.0.1, and patch2 for 43.0.2. For ESR you probably need the same, but if you felt very comfortable with patch 1 you could consider doing an 38.5.0esr build2 with it, and then 38.5.1 with patch2.
Flags: needinfo?(nthomas)
Nick, as a consistency (and to avoid further confusions), I would like to follow the same plan as release.
Last I asked the sha1 deprecation was not going to be backported to esr.
Flags: needinfo?(robert.strong.bugs)
(In reply to Nick Thomas [:nthomas] from comment #141)
> rstrong, can we use telemetry to look at how many times we don't use the
> service for Nightly updates, and general failure rates ? Particularly if we
> get buildid information.
It is in telemetry and we saw an uptick in errors the other day as a matter of fact
http://alerts.telemetry.mozilla.org/index.html#/detectors/1/metrics/1235/alerts/?from=2015-12-03&to=2015-12-03
(In reply to Robert Strong [:rstrong] (use needinfo to contact me) from comment #144)
> Last I asked the sha1 deprecation was not going to be backported to esr.
FYI regarding esr38 getting sha1 deprecation backported

Bug 1116409 comment #89

So, I don't think this needs to be backported to esr

needinfo'ing sylvestre since if it is going to be backported there are other patches that would need to be backported.
Flags: needinfo?(sledru)
Depends on: 1232078
I am not sure to understand why David Keeler say it won't. Is it because we don't want to or because it won't affect ESR?
Flags: needinfo?(sledru)
No idea and best to ask David.

David, Sylvestre would like to know why the sha-1 deprecation code won't be backported to esr38 ( see Bug 1116409 comment #89 ).
Flags: needinfo?(dkeeler)
Sorry - I wasn't clear. In bug 1116409 comment 89 I was referring to the TLS certificate verification code in Firefox itself, not platform-specific code-singing details. (As for why that won't be backported to esr38, it's a combination of the work it would take to do that and that if my reading of https://wiki.mozilla.org/RapidRelease/Calendar#Future_branch_dates is correct, esr38 will be EOL before it really makes a difference.)
Flags: needinfo?(dkeeler)
Robert, Seems like we are talking about something different, right? so, we have to take, right?
Flags: needinfo?(robert.strong.bugs)
As I understand it from mhowell, this issue will affect not just XP service pack 2 users who are either updating Firefox or downloading it. It also affects Windows users using SHA-2.  I think that means we need this in ESR. Matt or Robert should be able to explain this though.
FTR. We still need to figure out plans for aurora and beta. Possible scenarios:

Aurora:

Current status:
a) Aurora nightly updates are frozen to the version before merge (Monday edition)
b) All patches are uplifted

What we should do:
1) Backout the second patch
2) Updates will be enabled on Friday
3) Land the second patch on Sunday to get it in Monday's nightly builds
4) Add a watershed rule in Balrog, similar to mozilla-central


Beta:

Current status:
1) None of the patches are uplifted

What we should do:
1) Land the first patch now. It'll be in 44.0b1
2) Land the second patch after we ship 44.0b1
3) Add a watershed rule in Balrog


Matt, Nick: does it sound sane?
Flags: needinfo?(nthomas)
Flags: needinfo?(mhowell)
(In reply to Rail Aliiev [:rail] from comment #152)
> Matt, Nick: does it sound sane?

Yes, that sounds to me as if it would work. Thanks for laying that all out.
Flags: needinfo?(mhowell)
(In reply to Sylvestre Ledru [:sylvestre] from comment #150)
> Robert, Seems like we are talking about something different, right? so, we
> have to take, right?
It is different but without doing the testing myself I don't know whether this is necessary to take on esr38 and defer to mhowell on that.
Flags: needinfo?(robert.strong.bugs)
Hi Matt, based on comment 152, would you be able to request patch1 for uplift to mozilla-beta? I can then work with sheriffs to get it landed before I gtb 44.0b1.
Flags: needinfo?(mhowell)
Comment on attachment 8697027 [details] [diff] [review]
Part 1: Inform the updater and maintenance service of the new SHA2 certicicate issuer

Approval Request Comment
[Feature/regressing bug #]: This bug and bug 1226191
[User impact if declined]: Potential install and update failures due to OS security prompts
[Describe test coverage new/current, TreeHerder]: Patch received manual QA from SoftVision; see comment 134 and earlier comments.
[Risks and why]: Very low due to extensive testing and the passive nature of this part of the change; this patch only does setup for part 2, which actually flips the switch.
[String/UUID change made/needed]: None
Flags: needinfo?(mhowell)
Attachment #8697027 - Flags: approval-mozilla-beta?
Comment on attachment 8697027 [details] [diff] [review]
Part 1: Inform the updater and maintenance service of the new SHA2 certicicate issuer

Given that this patch has been tested by SV (and others), it seems safe and necessary to take this in beta44 before beta1 goes live. Approved.
Attachment #8697027 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
(In reply to Ritu Kothari (:ritu) from comment #157)
> Comment on attachment 8697027 [details] [diff] [review]
> Part 1: Inform the updater and maintenance service of the new SHA2
> certicicate issuer
> 
> Given that this patch has been tested by SV (and others), it seems safe and
> necessary to take this in beta44 before beta1 goes live. Approved.

reminders to releng to update aurora, beta, and release watershed rules.


dates of when we need extra rule by:

release channel:
    * 43.0 (no sha2 patch) -> 43.0.1 (sha2 patch 1) -> 43.0.1 (sha2 patch 2)
    * when we need watershed rule: 43.0.1 is starting tomorrow

beta channel:
    * 43.0b10 RC (no sha2 patch) -> 44.0b1 (sha2 patch 1) -> 44.0b2 (sha2 patch 2)
    * when we need watershed rule: 44.0b1 is starting today/tomorrow


aurora channel (currently locked till friday):
    * 44.0 nightly current lock (no sha2 patches?) -> 45.0 friday nightly (sha2 patch 1 only) -> 45.0 sunday nightly (sha2patch 2)
    * is patch 2 backed out of aurora yet?
    * when we need watershed rule: before friday ideally


am I on the same page?
Flags: needinfo?(rail)
Flags: needinfo?(jlund)
We can create watershed rules whenever we have builds with patch1 (need their build IDs).

release: after we ship 43.0.1
beta: after we ship beta1
aurora: after QE sign offs the Friday's nightly
Flags: needinfo?(rail)
I concur with comment #158, as modified by #159. One slight tweak - beta channel people will mostly be coming from 43.0 build1 rather than 43.0b<something>.

I attempted to backout part two (mc: b40ba117fa75) on aurora, but the tree is closed at the moment for test failures.
Flags: needinfo?(nthomas)
(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #151)
> As I understand it from mhowell, this issue will affect not just XP service
> pack 2 users who are either updating Firefox or downloading it. It also
> affects Windows users using SHA-2.  I think that means we need this in ESR.
> Matt or Robert should be able to explain this though.

This is my understanding as well. I don't know where this "we don't need it on ESR" idea came from, but AIUI we need it on every release channel that we want to keep updating after Dec 31st, which would seem to me to be all of them (given that people on ESR 38 want to be able to update to ESR 45). And what's more, comment 92 suggests that we need everyone to have the first part of the patch before Dec 31st. Given that it's now Dec 16th, and our update rates are not stellar, it seems like we aren't in a good place here, either for ESR or the normal release :-(

Am I right, or have I missed something that makes everything OK?

Gerv
Our next step here is to land both patches on aurora, doing a build in between, I believe making sure that users are forced to update to a build with only patch 1, then to the next build.   Jordan, Nick, does that sound right?  

mozilla-aurora is closed right now, but I'd like to land this as soon as possible anyway. Wes can you land the first patch on m-a?  We need to do this soon, because we have to bring it all the way up to release and esr.
Flags: needinfo?(wkocher)
Flags: needinfo?(nthomas)
Comment on attachment 8696728 [details] [diff] [review]
Part 2: Use SHA-2 to sign Windows binaries, and switch the stub installer to expect that

part2 backout on aurora: https://hg.mozilla.org/releases/mozilla-aurora/rev/4e7f5f9b7bd0


Current state is ...

Branch                part1           part2
mozilla-central       9aa2dae27ae5    b40ba117fa75
mozilla-aurora        9aa2dae27ae5    backed out
mozilla-beta          cdcd29e5c79e    not landed
mozilla-release       not landed      not landed
Flags: needinfo?(nthomas)
Updated with esr38:

Branch                part1           part2
mozilla-central       9aa2dae27ae5    b40ba117fa75
mozilla-aurora        9aa2dae27ae5*   backed out     (* uplifted by merge)
mozilla-beta          cdcd29e5c79e    not landed
mozilla-release       not landed      not landed
mozilla-esr38         not landed      not landed
Which brings us to:

Branch                part1           part2
mozilla-central       9aa2dae27ae5    b40ba117fa75
mozilla-aurora        9aa2dae27ae5*   backed out     (* uplifted by merge)
mozilla-beta          cdcd29e5c79e    not landed
mozilla-release       3f46faf48a4a    not landed
mozilla-esr38         07df110a15c8    not landed
Comment on attachment 8697027 [details] [diff] [review]
Part 1: Inform the updater and maintenance service of the new SHA2 certicicate issuer

Approved to land on ESR38 branch (on behalf of Liz)
Attachment #8697027 - Flags: approval-mozilla-esr38+
Comment on attachment 8697027 [details] [diff] [review]
Part 1: Inform the updater and maintenance service of the new SHA2 certicicate issuer

We need this on m-r as well, a+ on Liz's behalf.
Attachment #8697027 - Flags: approval-mozilla-release+
Matt, could you please nominate patch2 for uplift to m-b and m-r? I would like to get this patch landed on m-b today (if possible) so it's included in 44.0b2. Thanks!
Flags: needinfo?(mhowell)
Ditto for esr :)
Comment on attachment 8696728 [details] [diff] [review]
Part 2: Use SHA-2 to sign Windows binaries, and switch the stub installer to expect that

Release43+, Beta44+
Attachment #8696728 - Flags: approval-mozilla-release+
Attachment #8696728 - Flags: approval-mozilla-beta+
Comment on attachment 8696728 [details] [diff] [review]
Part 2: Use SHA-2 to sign Windows binaries, and switch the stub installer to expect that

beta and release are already approved, but I'll request ESR uplift here; let me know if you need more.

[Approval Request Comment]
If this is not a sec:{high,crit} bug, please state case for ESR consideration: Without this patch, future updates will not pass code signature checks and will either annoy users with a UAC prompt or may fail to apply for non-administrator users.

User impact if declined: 
See above.

Fix Landed on Version:
Release 43.0.2

Risk to taking this patch (and alternatives if risky): 
Very low due to extensive testing by SoftVision.

String or UUID changes made by this patch: 
None.

See https://wiki.mozilla.org/Release_Management/ESR_Landing_Process for more info.
Flags: needinfo?(mhowell)
Attachment #8696728 - Flags: approval-mozilla-esr38?
Depends on: 1233779
Unless someone complains, I plan to proceed with mozilla-aurora as stated below:

Current status: patch1 is laned

* Land patch2 on Sunday afternoon to get it into Monday nightly build
* lock updates to Sunday's nightly (patch1)
* Add a watershed
* Let QE test updates on the auroratest channel on Monday (with patch2 landed) 
* if everything looks OK, I'm going to unlock the updates again
(In reply to Rail Aliiev [:rail] from comment #175)
> Unless someone complains, I plan to proceed with mozilla-aurora as stated
> below:
> 
> Current status: patch1 is laned
> 
> * Land patch2 on Sunday afternoon to get it into Monday nightly build
> * lock updates to Sunday's nightly (patch1)
> * Add a watershed
> * Let QE test updates on the auroratest channel on Monday (with patch2
> landed) 
> * if everything looks OK, I'm going to unlock the updates again

The tree is still closed. :/ I'll wait until it's open.
Flags: needinfo?(jlund)
Depends on: 1234273
Depends on: 1234277
I pushed it to aurora in https://hg.mozilla.org/releases/mozilla-aurora/rev/3556870721fb and backed out in https://hg.mozilla.org/releases/mozilla-aurora/rev/454879ae971d

We want to wait until bouncer is ready for this.
We have a choice here to keep this on beta 2 or back it out. Ritu what do you think? It would affect XP sp2 users coming to download a fresh beta.  It doesn't affect people updating on beta.
Flags: needinfo?(rkothari)
Rail, could you please backout patch2 from Beta2 as well? We can re-land patch2 in next Beta i.e. Beta4 (as I will skip Beta3). This will give us enough time to put in place the rules to direct xp users to install 44.0b1/44.0b2.
Flags: needinfo?(rkothari) → needinfo?(rail)
Flags: needinfo?(rail)
Blocks: 1234339
Comment on attachment 8696728 [details] [diff] [review]
Part 2: Use SHA-2 to sign Windows binaries, and switch the stub installer to expect that

Can someone land patch 2 in the esr38 branch? thanks
Attachment #8696728 - Flags: approval-mozilla-esr38? → approval-mozilla-esr38+
This has been added to the release notes with two wordings for 43.0.1 & 43.0.2 but also beta (only the second) and esr (38.5.1 & 38.5.2):
* Prepare to use SHA-256 signing certificate for Windows builds, to meet new signing requirement (1079858)
* Use a SHA-256 signing certificate for Windows builds, to meet new signing requirements
relnote-firefox: --- → 43+
The [Digital Signatures] tab of the [Properties] of firefox.exe and Firefox Setup 43.0.2.exe still claim that the digest algorithm is sha1. Is this expected?
I'm also seeing the SHA1 in the latest m-c Nightly/Tinderbox builds

Image here: 
http://i.imgur.com/myDjvox.jpg 

win7 x64
Ben, can you help with comment #185?
Flags: needinfo?(bhearsum)
Although the digest algorithm is sha1, the certificate chains to "DigiCert SHA2 Assured ID Code Signing CA".
(In reply to Rail Aliiev [:rail] from comment #186)
> According to
> https://hg.mozilla.org/releases/mozilla-release/file/FIREFOX_43_0_2_RELEASE/
> toolkit/mozapps/installer/signing.mk#l11 if should be sha2...

I'm trying to make the same changes with Fx China repack, IIUC, changes in build-tools[1] looks like "sha2signcode" only means switching to a sha2 cert, w/o using sha2 as the file digest algorithm; this MS technet article[2] also mentions "Code signature File Hashes: Microsoft does not require these file hashes to be done using SHA-2. Windows will also not enforce policies on these hashes. If pre-image attacks on SHA-1 become feasible we will reevaluate how the system trusts signatures made with such file hashes. ".

Maybe this means 43.0.2 is signed as intented?

The same article also says "Timestamp Signature Hashes: Windows will no longer trust files with the Mark of the Web attribute that are timestamped with a SHA-1 signature hash after 1/1/2017 on Windows 10 systems. Microsoft requires CAs to start issuing new timestamp signature hashes using only the SHA-2 algorithm after 1/1/2016. For developers targeting Windows Vista and Server 2008, CAs will be allowed to continue issuing SHA-1 timestamps. " which may means switching to a RFC 3161 time stamp server a year later?

[1]: https://github.com/mozilla/build-tools/commit/46f431bc869b88af166d4acdf30cfb5b8f6c0111
[2]: http://social.technet.microsoft.com/wiki/contents/articles/32288.windows-enforcement-of-authenticode-code-signing-and-timestamping.aspx#Enforcement_in_General
(In reply to Hector Zhao [:hectorz] from comment #190)
> Maybe this means 43.0.2 is signed as intented?

I hope so. Just wanted to double check with Ben who spent a lot of time digging into all details.
If we use sha1 for the digest algorithm, why don't we use multiple signatures? According to comment #11, it was ruled out because osslsigncode doesn't support mixing different hash algorithms. But we can use sha1 for both sha1 and sha2 certificates, no?
It will also fix the installer problem on XP SP2.
(In reply to Masatoshi Kimura [:emk] from comment #192)
> If we use sha1 for the digest algorithm, why don't we use multiple
> signatures? According to comment #11, it was ruled out because osslsigncode
> doesn't support mixing different hash algorithms. But we can use sha1 for
> both sha1 and sha2 certificates, no?
> It will also fix the installer problem on XP SP2.

Reading comment 24 ~ comment 31, I think there're concerns about multiple signatures support in some unpatched version of Win7?
Should not be a problem, unpatched versions of Win7 will not be aware of the deadline anyway. It would be sha1 and sha256 signatures, BTW.
(In reply to Rail Aliiev [:rail] from comment #191)
> (In reply to Hector Zhao [:hectorz] from comment #190)
> > Maybe this means 43.0.2 is signed as intented?
> 
> I hope so. Just wanted to double check with Ben who spent a lot of time
> digging into all details.

Yes, sha1 digest is expected.

(In reply to Hector Zhao [:hectorz] from comment #193)
> (In reply to Masatoshi Kimura [:emk] from comment #192)
> > If we use sha1 for the digest algorithm, why don't we use multiple
> > signatures? According to comment #11, it was ruled out because osslsigncode
> > doesn't support mixing different hash algorithms. But we can use sha1 for
> > both sha1 and sha2 certificates, no?
> > It will also fix the installer problem on XP SP2.
> 
> Reading comment 24 ~ comment 31, I think there're concerns about multiple
> signatures support in some unpatched version of Win7?

This is correct. Some versions of Windows Vista and 7 do not support multiple signatures, and it wasn't clear whether or not they would deprecate SHA-1 (I know Yuhong says differently in comment #194, but we weren't going to bank our solution on this without verifying it, which we never could). Versions of Windows that don't support multiple signatures only see the first one, so had we signed with a SHA-1 cert, then SHA-2, these older versions of Vista and 7 may have been unable to find a valid signature. If we signed with SHA-2, then SHA-1, XP SP2 would definitely have been unable to find a valid signature.
Flags: needinfo?(bhearsum)
(In reply to Ben Hearsum (:bhearsum) from comment #196)
> (In reply to Rail Aliiev [:rail] from comment #191)
> > (In reply to Hector Zhao [:hectorz] from comment #190)
> > > Maybe this means 43.0.2 is signed as intented?
> > 
> > I hope so. Just wanted to double check with Ben who spent a lot of time
> > digging into all details.
> 
> Yes, sha1 digest is expected.

For anyone concerned, the latest Microsoft document on SHA-1 deprecation confirms that SHA-1 digest algorithms are still fine. Quoting from http://social.technet.microsoft.com/wiki/contents/articles/32288.windows-enforcement-of-authenticode-code-signing-and-timestamping.aspx:
"Microsoft does not require these file hashes to be done using SHA-2. Windows will also not enforce policies on these hashes. If pre-image attacks on SHA-1 become feasible we will reevaluate how the system trusts signatures made with such file hashes."
(In reply to Ben Hearsum (:bhearsum) from comment #196)
> 
> This is correct. Some versions of Windows Vista and 7 do not support
> multiple signatures, and it wasn't clear whether or not they would deprecate
> SHA-1 (I know Yuhong says differently in comment #194, but we weren't going
> to bank our solution on this without verifying it, which we never could).

I think there's no need to worry about Vista, the technet article says "Windows (version 7 and higher)". And maybe it's possible to test things with the instructions from the article's "Testing" section?

> Versions of Windows that don't support multiple signatures only see the
> first one, so had we signed with a SHA-1 cert, then SHA-2, these older
> versions of Vista and 7 may have been unable to find a valid signature.

The same article also calls dual signing "the best option" for "a single executable needs to work on all Windows systems down to Windows Vista" where SHA-1 is the first signature in the example.
(In reply to Ben Hearsum (:bhearsum) from comment #196)
> Versions of Windows that don't support multiple signatures only see the
> first one, so had we signed with a SHA-1 cert, then SHA-2, these older
> versions of Vista and 7 may have been unable to find a valid signature.

Those versions will not be affected by the SHA-1 deprecation because they are (by definition) unpatched.
Blocks: 1238556
See Also: → 1240345
Is there anything more to do here?
status-b2g-v2.5: fixed → ---
status-firefox43: affected → fixed
Flags: needinfo?(mhowell)
(In reply to Ryan VanderMeulen [:RyanVM] from comment #201)
> Is there anything more to do here?

Bug 1232078 is still outstanding, but that's all I know of.
Flags: needinfo?(mhowell)
Any reason we can't close this bug?
Flags: needinfo?(mhowell)
I don't know of any; I hadn't done that yet because the actual work moved from me to falling under releng, but as far as I know, they've taken care of everything now. I'll go ahead and resolve it.
Status: ASSIGNED → RESOLVED
Closed: 9 years ago → 8 years ago
Flags: needinfo?(mhowell)
Resolution: --- → FIXED
Depends on: 1245842
Depends on: 1272925
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: