Affected versions

• 102.0a1
• 101.0b2
• 100.0

Affected platforms

• Windows 10
• macOS 11
• Ubuntu 20

Steps to reproduce

1. Launch Firefox using a new profile
2. Go to Add-ons Manager → Plugins tab
3. Wait several minutes for plugin installation
4. Try to play any streaming service (eg. Netflix)

Expected result

• The plugin is properly downloaded
• The message is no longer displayed
• The streaming service works

Actual result:

• The plugin is not installed
• The yellow warning is still display
• No streaming services seams to work

Regression range

• N/A

Other notes

• Forcing the Widevine update, browser restart, waiting a significant amount of time (~1h 30m), trying to play different content, toggling the “Play DRM-controlled content” checkbox, nothing seems to work once I am reproducing this issue.
• Not every test machine is affected after double checking with my colleagues. Personally, I’m reproducing the issue across platforms (Windows, Mac, Ubuntu) on all the channels.
• What seems odd is that my college reproduces the issue only on Nightly and Beta channel on Ubuntu 20 (RC is not affected), and using the same test machine, Win 10 has no problem. Moreover that specific Ubuntu 20 was used last week for the Nightly testing and everything worked fine.

Sounds like a failed initial install, and then we do not retry quickly afterward? We're doing some investigative work into the whole gmp install pipeline currently.

Jim, I need to verify some uplifts related to Netflix. Is there a way to manually install the Widevine plugin?

If you can please try this -

1. in a fresh profile before visiting netflix
2. visit about:config and enable gmp install logging by creating the numeric pref 'media.gmp.log.level' set the value to 5
3. open the browser console
4. visit netflix to view content and (hopefully) trigger the download UI
5. copy paste anything addon related that shows up in the console.

This is everything related to addons in the browser console.

This was provided from the latest Nightly on Windows 10.

Looks like the install of the cdm file failed for some reason. We downloaded it into a temp folder and unpacked it, but when Firefox tried to copy it into the profile dir something went wrong. Your about support says you're running 'Cisco AMP for Endpoints' for malware protection, does that software have any logs in it that might indicate it intervened?

No, that's definitely not it. The station I'm currently using does not have any AV installed. This issue also occurs on machines that do not have Cisco AMP installed.

Thanks for the report. I can repro. Investigating.

Notes

• This is happening for me on versions of nightly going back at least as far as the start of April.
• This also impacts beta.

I'm confident this is a recent regression, but since it's impacting early April releases of nightly, it may be something outside the client.

Think I've got it. Widevine have re-uploaded the zips with the files inside having updated metadata (dates). This changes the binary for the zip, which changes the hashes for the zips, we then reject the zips because they have a bad hash (our code working as intended to defend against MiTM attacks).

We can fix this in our balrog rules, and I'll move to do so now.

Is this really an S1 given that we haven't started rolling out this version to release users yet?

I think it has a good priority and severity considering it blocks some parts of testing the PiP and definitely others.

(In reply to Ryan VanderMeulen [:RyanVM] from comment #11)

Is this really an S1 given that we haven't started rolling out this version to release users yet?

Let's back it off to S2 now that we're confident is doesn't impact release.

I think this should also be resolved now by rule updates made in bug 1758423.

This issue is no longer reproducible on Windows 10 and Ubuntu 20.04 (double checked with my colleagues on the previously affected test machines).
But, we can still reproduce it on macOS, on Nightly and Beta channels, tested with OS versions 11.5 and 10.15.

Confirming we're still having an issue on Mac in Nightly.

1652106967625 addons.xpi WARN Checking /Applications/Firefox Nightly.app/Contents/Resources/distribution/extensions for addons
1652106968027 Toolkit.GMP TRACE GMPProvider.startup - enabled=true, gmpPath=/Users/mozilla/Library/Application Support/Firefox/Profiles/6htx2p7s.test widevine/gmp-gmpopenh264/1.8.1.1
1652106968027 Toolkit.GMP INFO GMPProvider.startup - adding gmp directory /Users/mozilla/Library/Application Support/Firefox/Profiles/6htx2p7s.test widevine/gmp-gmpopenh264/1.8.1.1
1652106968028 Toolkit.GMP TRACE GMPProvider.startup - enabled=true, gmpPath=null
1652106968028 Toolkit.GMP INFO GMPProvider.startup - adding clearkey CDM directory /Applications/Firefox Nightly.app/Contents/Resources/gmp-clearkey/0.1
BackgroundUpdate: _reasonsToNotScheduleUpdates: Failed to check for Maintenance Service Registry Key: [Exception... "Component returned failure code: 0x80004001 (NS_ERROR_NOT_IMPLEMENTED) [nsIUpdateProcessor.getServiceRegKeyExists]" nsresult: "0x80004001 (NS_ERROR_NOT_IMPLEMENTED)" location: "JS frame :: resource://gre/modules/BackgroundUpdate.jsm :: _reasonsToNotScheduleUpdates :: line 241" data: no] BackgroundUpdate.jsm:243
debuggee 'resource://devtools/shared/loader/base-loader.js:289' would run 5 builtin-modules.js:206:11
debuggee 'resource://devtools/shared/loader/base-loader.js:289' would run 2 builtin-modules.js:206:11
MediaKeySystemAccess::GetKeySystemStatus(com.widevine.alpha) result=cdm-not-installed msg='CDM is not installed'
1652106994073 Toolkit.GMP INFO GMPInstallManager.simpleCheckAndInstall Last check was: 1652106994 seconds ago, minimum seconds: 86400
1652106994074 Toolkit.GMP INFO GMPInstallManager._getURL Using url: https://aus5.mozilla.org/update/3/GMP/%VERSION%/%BUILD_ID%/%BUILD_TARGET%/%LOCALE%/%CHANNEL%/%OS_VERSION%/%DISTRIBUTION%/%DISTRIBUTION_VERSION%/update.xml
1652106994079 Toolkit.GMP INFO GMPInstallManager._getURL Using url (with replacement): https://aus5.mozilla.org/update/3/GMP/102.0a1/20220509094710/Darwin_x86_64-gcc3/en-US/nightly/Darwin%2021.4.0/default/default/update.xml
1652106994079 Toolkit.GMP INFO GMPInstallManager.checkForAddons Fetching product addon list url=https://aus5.mozilla.org/update/3/GMP/102.0a1/20220509094710/Darwin_x86_64-gcc3/en-US/nightly/Darwin%2021.4.0/default/default/update.xml, allowNonBuiltIn=true, certs=null, checkContentSignature=true
1652106994423 Toolkit.GMP INFO GMPAddon.constructor Created new addon: gmp-gmpopenh264 (isValid: true, isInstalled: true, hashFunction: sha512, hashValue: fc1ddb4b7cff2f27a0f10d033850211d9186ae7576194e1387974e8f7a8c350ef50e12238694003f16ed5917ffcd22ae0d54cfb738b9724440d65f8afdf7a49f, size: 466258)
1652106994423 Toolkit.GMP INFO GMPAddon.constructor Created new addon: gmp-widevinecdm (isValid: true, isInstalled: false, hashFunction: sha512, hashValue: 7d2b92552c137d8cbbbbe97047c7e409859ad01107ea2a4895f4035ebf876efac050dc2868435fa70db3c422344e36db148a0f2c6f2f9b319919284152e2b033, size: 7213441)
1652106994424 Toolkit.GMP INFO GMPInstallManager.simpleCheckAndInstall Found 2 addons advertised.
1652106994424 Toolkit.GMP INFO GMPInstallManager.simpleCheckAndInstall Found addon: gmp-gmpopenh264 (isValid: true, isInstalled: true, hashFunction: sha512, hashValue: fc1ddb4b7cff2f27a0f10d033850211d9186ae7576194e1387974e8f7a8c350ef50e12238694003f16ed5917ffcd22ae0d54cfb738b9724440d65f8afdf7a49f, size: 466258)
1652106994424 Toolkit.GMP INFO GMPInstallManager.simpleCheckAndInstall Addon |gmp-gmpopenh264| already installed.
1652106994424 Toolkit.GMP INFO GMPInstallManager.simpleCheckAndInstall Found addon: gmp-widevinecdm (isValid: true, isInstalled: false, hashFunction: sha512, hashValue: 7d2b92552c137d8cbbbbe97047c7e409859ad01107ea2a4895f4035ebf876efac050dc2868435fa70db3c422344e36db148a0f2c6f2f9b319919284152e2b033, size: 7213441)
MediaKeySystemAccess::GetKeySystemStatus(com.widevine.alpha) result=cdm-not-installed msg='CDM is not installed'
1652106995163 Toolkit.GMP INFO GMPInstallManager.simpleCheckAndInstall Last check was: 1652106995 seconds ago, minimum seconds: 86400
1652106995163 Toolkit.GMP INFO GMPInstallManager._getURL Using url: https://aus5.mozilla.org/update/3/GMP/%VERSION%/%BUILD_ID%/%BUILD_TARGET%/%LOCALE%/%CHANNEL%/%OS_VERSION%/%DISTRIBUTION%/%DISTRIBUTION_VERSION%/update.xml
1652106995163 Toolkit.GMP INFO GMPInstallManager._getURL Using url (with replacement): https://aus5.mozilla.org/update/3/GMP/102.0a1/20220509094710/Darwin_x86_64-gcc3/en-US/nightly/Darwin%2021.4.0/default/default/update.xml
1652106995163 Toolkit.GMP INFO GMPInstallManager.checkForAddons Fetching product addon list url=https://aus5.mozilla.org/update/3/GMP/102.0a1/20220509094710/Darwin_x86_64-gcc3/en-US/nightly/Darwin%2021.4.0/default/default/update.xml, allowNonBuiltIn=true, certs=null, checkContentSignature=true
1652106995221 Toolkit.GMP INFO GMPAddon.constructor Created new addon: gmp-gmpopenh264 (isValid: true, isInstalled: true, hashFunction: sha512, hashValue: fc1ddb4b7cff2f27a0f10d033850211d9186ae7576194e1387974e8f7a8c350ef50e12238694003f16ed5917ffcd22ae0d54cfb738b9724440d65f8afdf7a49f, size: 466258)
1652106995221 Toolkit.GMP INFO GMPAddon.constructor Created new addon: gmp-widevinecdm (isValid: true, isInstalled: false, hashFunction: sha512, hashValue: 7d2b92552c137d8cbbbbe97047c7e409859ad01107ea2a4895f4035ebf876efac050dc2868435fa70db3c422344e36db148a0f2c6f2f9b319919284152e2b033, size: 7213441)
1652106995222 Toolkit.GMP INFO GMPInstallManager.simpleCheckAndInstall Found 2 addons advertised.
1652106995222 Toolkit.GMP INFO GMPInstallManager.simpleCheckAndInstall Found addon: gmp-gmpopenh264 (isValid: true, isInstalled: true, hashFunction: sha512, hashValue: fc1ddb4b7cff2f27a0f10d033850211d9186ae7576194e1387974e8f7a8c350ef50e12238694003f16ed5917ffcd22ae0d54cfb738b9724440d65f8afdf7a49f, size: 466258)
1652106995222 Toolkit.GMP INFO GMPInstallManager.simpleCheckAndInstall Addon |gmp-gmpopenh264| already installed.
1652106995222 Toolkit.GMP INFO GMPInstallManager.simpleCheckAndInstall Found addon: gmp-widevinecdm (isValid: true, isInstalled: false, hashFunction: sha512, hashValue: 7d2b92552c137d8cbbbbe97047c7e409859ad01107ea2a4895f4035ebf876efac050dc2868435fa70db3c422344e36db148a0f2c6f2f9b319919284152e2b033, size: 7213441)
debuggee 'resource://devtools/shared/loader/base-loader.js:289' would run 2 builtin-modules.js:206:11

1652106995848 Toolkit.GMP ERROR GMPInstallManager.simpleCheckAndInstall Could not check for addons: {"status":"failed","results":[{"id":"gmp-widevinecdm","result":"failed"}]} Log.jsm:723

1652106995852 Toolkit.GMP ERROR GMPInstallManager.simpleCheckAndInstall Could not check for addons: {"status":"failed","results":[{"id":"gmp-widevinecdm","result":"failed"}]} Log.jsm:723

This appears to be due to the Mac64 zip being changed following our previous update to the hashes. We've bumped the hash again to reflect this. I've just tested MacOS64 following this and think we're good, but it would be useful to know that other folks are also seeing this resolved before I close this again.

Installation seems to work fine for us, at this moment, across platforms (Windows, Ubuntu, macOS) on all channels. We’ll keep an eye on, these days, report back if we’ll run into any problem.

Per comment 17. Please reopen if the issue returns.

We didn't encountered any installation problem since my last comment, while testing across platform, on all channels, therefore I will mark this as verified fixed.