Found in

• 102

Affected versions

• 102/103

Affected platforms

• Windows 10
• macOS 11.5/13
• Ubuntu 20.04/22.04

Steps to reproduce

1. Install an older Firefox (version 102)
2. Add the Update Pinning policy inside the Firefox installation folder with the pin version set to 102
3. Open the browser and confirm that the Update Pinning policy has an active status inside about:policies
4. Attempt to update the browser

Expected result

• Firefox can’t be updated, it remains to the pinned version (102)

Actual result

• Firefox updates to 103.0.2.

Regression range

• N/A

Additional notes

• Once the policy is removed the browser is updated to the latest version available (104)
• Update Pinning policy seemed to work until this morning, a complete round of tests were executed yesterday on macOS without any issue. As for Windows 10, we saw that the very first time when setting the policy on a build will not actually respect it (it does appear in about:policy) and update past the pin (repeating a second time and so on will respect the pin every time). As of today the pin is not respected at all.
• ESR respects the pin policy, no issue on this channel.

This feels like it might be something on the server?

Can you attach about:policies and the update log?

I somewhat disagree with the "Expected Results" here:

Expected result

• Firefox can’t be updated, it remains to the pinned version (102)

I believe that it should be able to be upgraded to 102.0.1, since that is still version 102. I agree that it should not, however, be able to update to 103.0.2. But I cannot actually reproduce this behavior.

I downloaded this Firefox installer and used this policy file:

{
  "policies": {
    "AppUpdatePin": "102."
  }
}

Initially, my update URL is this expected URL which only provides an update to 102.0.1. When I use the update UI, it updates to 102.0.1. After restarting, my update URL is this expected URL which provides no available updates. As expected, the update UI shows that Firefox is up to date.

I'm seeing the same behavior as reported.

I started a Windows Sandbox, installed Firefox, added the policies.json (what you had above) before started it for the first time, started it and checked for updates and it updated to 103.0.2.

I'll do some logging.

Here's the full log. I set app.update.log in policy so it happened early in startup.

1661456860498	addons.xpi	WARN	Checking C:\Users\WDAGUtilityAccount\FIREFOX\distribution\extensions for addons
Error: Can't find profile directory. XULStore.jsm:66:15
SearchSettings: get: No settings file exists, new profile? DOMException: Could not open the file at C:\Users\WDAGUtilityAccount\AppData\Roaming\Mozilla\Firefox\Profiles\nlb5ciou.default-release\search.json.mozlz4 SearchSettings.jsm:108:18
AUS:SVC Checker:getUpdateURL - update URL: https://aus5.mozilla.org/update/6/Firefox/102.0/20220623063721/WINNT_x86_64-msvc-x64/en-US/release/Windows_NT%2010.0.0.0.22000.856%20(x64)/ISET:SSE4_2,MEM:4095/default/default/update.xml?pin=102.
AUS:SVC UpdateService.canUsuallyCheckForUpdates - able to check for updates
AUS:SVC UpdateService.canCheckForUpdates - able to check for updates
AUS:SVC Checker: checkForUpdates, force: false
AUS:SVC Creating UpdateService
AUS:SVC Logging current UpdateService status:
AUS:SVC UpdateService.canUsuallyCheckForUpdates - able to check for updates
AUS:SVC UpdateService.canCheckForUpdates - able to check for updates
AUS:SVC getCanApplyUpdates - testing write access C:\ProgramData\Mozilla-1de4eec8-1241-4177-a864-e594e8d1fb38\updates\5FF346744023211F\update.test
AUS:SVC isServiceInstalled - returning true
AUS:SVC shouldUseService - returning true
AUS:SVC getCanApplyUpdates - bypass the write checks because the Windows Maintenance Service can be used
AUS:SVC isServiceInstalled - returning true
AUS:SVC shouldUseService - returning true
AUS:SVC getCanStageUpdates - able to stage updates using the service
AUS:SVC Elevation required: false
AUS:SVC Other instance of the application currently running: false
AUS:SVC Downloading: false
AUS:SVC End of UpdateService status
AUS:SVC UpdateService.canUsuallyCheckForUpdates - able to check for updates
AUS:SVC UpdateService.canCheckForUpdates - able to check for updates
AUS:SVC waitForOtherInstances - beginning polling
AUS:SVC waitForOtherInstances - no other instances found, exiting
AUS:SVC Checker:getUpdateURL - update URL: https://aus5.mozilla.org/update/6/Firefox/102.0/20220623063721/WINNT_x86_64-msvc-x64/en-US/release/Windows_NT%2010.0.0.0.22000.856%20(x64)/ISET:SSE4_2,MEM:4095/default/default/update.xml?pin=102.
AUS:SVC Checker:checkForUpdates - sending request to: https://aus5.mozilla.org/update/6/Firefox/102.0/20220623063721/WINNT_x86_64-msvc-x64/en-US/release/Windows_NT%2010.0.0.0.22000.856%20(x64)/ISET:SSE4_2,MEM:4095/default/default/update.xml?pin=102.
Error: Could not get children of file(C:\Users\WDAGUtilityAccount\AppData\Roaming\Mozilla\Firefox\Profiles\nlb5ciou.default-release\thumbnails) because it does not exist PromiseWorker.jsm:106
Error: Can't find profile directory. 4 XULStore.jsm:66:15
AUS:SVC Checker:onLoad - request completed downloading document
AUS:SVC Checker:onLoad - Getting sslStatus failed.
AUS:SVC Checker:onLoad - number of updates available: 1
AUS:SVC getCanApplyUpdates - testing write access C:\ProgramData\Mozilla-1de4eec8-1241-4177-a864-e594e8d1fb38\updates\5FF346744023211F\update.test
AUS:SVC isServiceInstalled - returning true
AUS:SVC shouldUseService - returning true
AUS:SVC getCanApplyUpdates - bypass the write checks because the Windows Maintenance Service can be used
AUS:SVC UpdateService:_selectAndInstallUpdate - download the update
AUS:SVC Creating Downloader
AUS:SVC UpdateService:_downloadUpdate
AUS:SVC getCanUseBits - BITS can be used to download updates
AUS:SVC Downloader:_canUseBits - Patch is able to use BITS download
AUS:SVC Downloader:downloadUpdate - Starting BITS download with url: https://download.mozilla.org/?product=firefox-103.0.2-partial-102.0&os=win64&lang=en-US, updateDir: C:\ProgramData\Mozilla-1de4eec8-1241-4177-a864-e594e8d1fb38\updates\5FF346744023211F\updates\downloading, filename: update.mar
AUS:SVC Downloader:downloadUpdate - BITS download running. BITS ID: {9E45B488-2797-4892-AF45-464FE6B38C1E}
UTM:SVC TimerManager:registerTimer - timerID: region-update-timer interval: 604800 skipFirst: false
UTM:SVC TimerManager:registerTimer - timerID: recipe-client-addon-run interval: 21600 skipFirst: false
AUS:SVC UpdateService.canUsuallyCheckForUpdates - able to check for updates
AUS:SVC isServiceInstalled - returning true
AUS:SVC shouldUseService - returning true
AUS:SVC getCanStageUpdates - able to stage updates using the service
AUS:SVC getCanUseBits - BITS can be used to download updates
Error: Can't find profile directory. XULStore.jsm:66:15
Key event not available on some keyboard layouts: key=“i” modifiers=“accel,alt,shift” id=“key_browserToolbox” browser.xhtml
AUS:SVC Downloader:onProgress - progress: 5767168/14015502
[Exception... "Component returned failure code: 0x8000ffff (NS_ERROR_UNEXPECTED) [nsIPrefBranch.getStringPref]"  nsresult: "0x8000ffff (NS_ERROR_UNEXPECTED)"  location: "JS frame :: resource://activity-stream/lib/ASRouter.jsm :: _remoteSettingsLoader :: line 284"  data: no] 2 ASRouter.jsm:284:28
AUS:SVC Downloader:onProgress - progress: 14015502/14015502
AUS:SVC Downloader:onStopRequest - downloader: BITS, status: 0
AUS:SVC Downloader:onStopRequest - status: 0, current fail: 0, max fail: 10, retryTimeout: 2000
AUS:SVC Downloader:_verifyDownload called
AUS:SVC Downloader:_verifyDownload downloaded size == expected size.
AUS:SVC isServiceInstalled - returning true
AUS:SVC shouldUseService - returning true
AUS:SVC isServiceInstalled - returning true
AUS:SVC shouldUseService - returning true
AUS:SVC getCanStageUpdates - able to stage updates using the service
AUS:SVC Downloader:onStopRequest - setting state to: pending-service
AUS:SVC isServiceInstalled - returning true
AUS:SVC shouldUseService - returning true
AUS:SVC getCanStageUpdates - able to stage updates using the service
AUS:SVC Downloader:onStopRequest - attempting to stage update: Firefox 103.0.2
AUS:SVC readStatusFile - status: applied, path: C:\ProgramData\Mozilla-1de4eec8-1241-4177-a864-e594e8d1fb38\updates\5FF346744023211F\updates\0\update.status
AUS:SVC readStringFromFile - file doesn't exist: C:\ProgramData\Mozilla-1de4eec8-1241-4177-a864-e594e8d1fb38\updates\5FF346744023211F\updates\0\bt.result
AUS:SVC readBinaryTransparencyResult - result: null, path: C:\ProgramData\Mozilla-1de4eec8-1241-4177-a864-e594e8d1fb38\updates\5FF346744023211F\updates\0\bt.result
AUS:SVC isServiceInstalled - returning true
AUS:SVC shouldUseService - returning true
UTM:SVC TimerManager:notify - notified timerID: browser-cleanup-thumbnails

Hmm, odd. I am currently getting the wrong XML (the one that advertises 103.0.2) when my query string doesn't include force=1. But if I add it, I get the correct XML. But I am sure that I tested the version without force=1 when I posted Comment 3 and got the correct thing. I'll look into it. But this definitely looks like a server-side problem to me. The Update URL appears to be correct, but the response is wrong.

Who should we talk to about the server issue?

I mean, I wrote the server code for this feature. So probably me. But I'll chat with the Balrog team about it too.

Initially, today we thought that this issue is reproducible only with RC 102.0, not with RC 102.0.1 (where, at first "Firefox is up to date" message is displayed as expected). But after several minutes, out of the blue the "Restart to Update Firefox" message replaced the "Firefox is up to date" one. The policy value we used is "102."
As for "I believe that it should be able to be upgraded to 102.0.1, since that is still version 102" you are right, I was ambiguously referring at RC 102 version as a whole. Firefox 102 upgraded to Fx 102.0.1 two days ago, before we ran into this behavior.

I believe that I've tracked down the problem. When a new version is first available, we intentionally slow the rollout of the new version. The process works like this:

1. A request for an update comes in.
2. We get the update rule that applies to that client which will have the main mapping (the newest version) and the fallback mapping (typically the version before that).
3. We check the "background rate" of the rule. 0% means we always serve the fallback mapping. 100% means we always serve the main mapping.
4. We "roll the dice" and compare that to the background rate to decide which we will return. This can be overridden if the request passes force=1, in which case the main mapping is always returned.

At this point, what should happen is for either chosen mapping to go through the pinning logic. But what happens instead is that the main mapping goes through the pinning logic, but if we choose the fallback mapping it gets returned right away.

I believe that the fix should be pretty easy. I'll start working on it right away.

I believe that the fix should be pretty easy. I'll start working on it right away.

FYI, I'm holding off on publishing docs for the policy until we get this figured out.

Kirk's PR is in balrog v3.20, which is now deployed to stage.
I've copied the 104.0.1 and 103.0.2 release blobs to balrog stage, and set up a rule on the release channel with mapping set to 104.0.1, fallback mapping to 103.0.2, and background rate 55. I've set a pinnable release with version 102..

Things are looking good now:

$ curl 'https://stage.balrog.nonprod.cloudops.mozgcp.net/update/6/Firefox/102.0/20220623063721/WINNT_x86_64-msvc-x64/en-US/release/Windows_NT%2010.0.0.0.22000.856%20(x64)/ISET:SSE4_2,MEM:4095/default/default/update.xml?force=1'
<?xml version="1.0"?>
<updates>
    <update actions="showURL" appVersion="104.0.1" buildID="20220829141339" detailsURL="https://www.mozilla.org/en-US/firefox/104.0.1/releasenotes/" displayVersion="104.0.1" openURL="https://www.mozilla.org/firefox/104.0.1/whatsnew/?oldversion=%OLD_VERSION%" type="minor">
        <patch type="complete" URL="https://download.mozilla.org/?product=firefox-104.0.1-complete&amp;os=win64&amp;lang=en-US" hashFunction="sha512" hashValue="9bef550ea9f67b03d1b0434443e1f5a1f771040d0db21d49078114d79428bcdec95a34fbc8eabcb4364346e87086707c41badcce7f7a1bc3f78845717e619d5f" size="60474119"/>
    </update>
</updates>
$ curl 'https://stage.balrog.nonprod.cloudops.mozgcp.net/update/6/Firefox/102.0/20220623063721/WINNT_x86_64-msvc-x64/en-US/release/Windows_NT%2010.0.0.0.22000.856%20(x64)/ISET:SSE4_2,MEM:4095/default/default/update.xml?force=-1'
<?xml version="1.0"?>
<updates>
    <update actions="showURL" appVersion="103.0.2" buildID="20220808125904" detailsURL="https://www.mozilla.org/en-US/firefox/103.0.2/releasenotes/" displayVersion="103.0.2" openURL="https://www.mozilla.org/firefox/103.0.2/whatsnew/?oldversion=%OLD_VERSION%" type="minor">
        <patch type="complete" URL="https://download.mozilla.org/?product=firefox-103.0.2-complete&amp;os=win64&amp;lang=en-US" hashFunction="sha512" hashValue="d9a143e0ac852373a1bb98d635ad3d055c3380d4edad3cf84f60de3d4c0b45e1d9e6bd1c03b19f3dcac802fbd2bb799a534619e69986b5d7ac34c11123705589" size="60226567"/>
    </update>
</updates>
$ curl 'https://stage.balrog.nonprod.cloudops.mozgcp.net/update/6/Firefox/102.0/20220623063721/WINNT_x86_64-msvc-x64/en-US/release/Windows_NT%2010.0.0.0.22000.856%20(x64)/ISET:SSE4_2,MEM:4095/default/default/update.xml?pin=102.&force=-1'
<?xml version="1.0"?>
<updates>
</updates>
$ curl 'https://stage.balrog.nonprod.cloudops.mozgcp.net/update/6/Firefox/102.0/20220623063721/WINNT_x86_64-msvc-x64/en-US/release/Windows_NT%2010.0.0.0.22000.856%20(x64)/ISET:SSE4_2,MEM:4095/default/default/update.xml?pin=102.&force=1'
<?xml version="1.0"?>
<updates>
</updates>

I believe that this issue has now been fixed.

Verified the issue and the pin works correctly now. Setting "102.0." will update to 102.0.1. Tested on macOS 13, Windows 11 and Ubuntu 22.04. Thanks Kirk!