Affected versions

• Latest Nightly 86.0a1

Affected platforms

• Windows 10 64bit
• Windows 7 64bit

Unaffected platforms

• MacOS 11
• Ubuntu 18.04

Steps to reproduce

1. Install old Nightly form a few days or yesterday (I tried with 08.01.2021 or 13.01.2021)
2. Access about nightly so an update is triggered
3. Restart Nightly

Expected result

• Nightly restarts after updates are applied

Actual result

• Nightly does not restart and can't be opened anymore

Regression range

• I also restarted nightly yesterday after update and it worked just fine so it must be something that broke since yesterday

This is the content of last-update.log

Performing a staged update
PATCH DIRECTORY C:\ProgramData\Mozilla\updates\6F193CCC56814779\updates\0
INSTALLATION DIRECTORY C:\Program Files\Firefox Nightly
WORKING DIRECTORY C:\Program Files\Firefox Nightly\updated
Non-critical update staging error! Falling back to non-staged updates and exiting

Last nightly changelog: https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=77dc70a7ac251056a60abbcdb89c8db32a79a253&tochange=cf6956a5ec8e21896736f96237b1476c9d0aaf45

Bugs in this changelog: https://bugzilla.mozilla.org/buglist.cgi?bug_id=1686615%2C1686296%2C1680637%2C1682395%2C1589658%2C1684532%2C1685708%2C1686341%2C1686350%2C1665066%2C1686275%2C1684524%2C1611554%2C1686337%2C1668243%2C1686207%2C1683940%2C1686490%2C1671166%2C1684123%2C1682068%2C1677590%2C1686516%2C1683658%2C1577774%2C1579017%2C1686113%2C1686305%2C1640607%2C1686532%2C1684122%2C1684110%2C1686507%2C1686573%2C1686528%2C1686335%2C1686372%2C1674305%2C1681012%2C1664730%2C1685034%2C1686045%2C1640053

Found out that even if I download the .zip versions of Nightly from today (https://hg.mozilla.org/mozilla-central/rev/cf6956a5ec8e21896736f96237b1476c9d0aaf45) it will not start so It seems it is not quite Update related.

Note:
On one Windows 10 machine in particular the build works just fine (1 out of 6 windows machines tested did not had this issue).

I can't reproduce on my windows 10 machine with the latest 64 bits en-US build. Are the builds you have the problem with localized builds maybe?

(In reply to Pascal Chevrel:pascalc from comment #4)

I can't reproduce on my windows 10 machine with the latest 64 bits en-US build. Are the builds you have the problem with localized builds maybe?

I have this issue with en-US, zh-CN, ro and others so it's not related to locales but I isolated this to Softvision machines :/ On my personal PC I have no issues. Will have to take this with IT department to find out more, could be related to Cisco AMP.

Ok, I don't have other reports of the bug, I am reenabling updates and will monitor our nightly communication channels to make sure this issue is indeed isolated to your environment.

Bug 1684532 in the regression range could be related if it's tied to something in SV's machine configuration.

Depending on Cisco AMP's implementation, but Bug 1684532 could be related because it blocks a kind of DLL injection. Unlike bug 1682304, Bug 1684532 becomes active only in Nightly. For further investigation, we need to know how Cisco AMP interacts with Firefox.

Someone from Softvision IT checked the logs of Cisco AMP on my PC and found this:

An attack was prevented in kernel32.dll at base address 0x00007FF94CEE0000 inside the firefox.exe process.
File signed by Mozilla Corporation with certificate serial 0 from DigiCert SHA2 Assured ID Code Signing CA.
Expires NaN:NaN:NaN, NaN 0NaN UTC. The certificate was warn trusted

They reached out to Cisco to analyze this.

As a workaround running the .exe with troubleshoot compat will make it work. (Windows compatibility mode: Windows 8 running on Win10)

Hi,
I'm working on windows 10. On the eleventh, I set my nightly profile to not update automatically and I hadn't updated since then. Today I went to help -> about firefox and updated FF from there. I told FF to restart for the update to be effective, FF closed itself but never opened again. I wasn't able to open it after either. I had to uninstall it and install an older version (from the sixth of January) and block updates.

Can I help?

Do the instructions in comment 10 work for you?

(In reply to Florencia Di Ciocco from comment #11)

Hi,
I'm working on windows 10. On the eleventh, I set my nightly profile to not update automatically and I hadn't updated since then. Today I went to help -> about firefox and updated FF from there. I told FF to restart for the update to be effective, FF closed itself but never opened again. I wasn't able to open it after either. I had to uninstall it and install an older version (from the sixth of January) and block updates.

In the working version of Nightly, could you set browser.enableAboutThirdParty to true on about:config, go to about:support, and see what are listed in the "Third-Party Modules" section at the bottom of the about:support page? That section shows modules which are injected into Firefox by third-party applications. If the patch for Bug 1684532 causes a launching problem, there should be an injected module which does not work well with our security feature.

(In reply to Toshihito Kikuchi [:toshi] from comment #13)

(In reply to Florencia Di Ciocco from comment #11)

Hi,
I'm working on windows 10. On the eleventh, I set my nightly profile to not update automatically and I hadn't updated since then. Today I went to help -> about firefox and updated FF from there. I told FF to restart for the update to be effective, FF closed itself but never opened again. I wasn't able to open it after either. I had to uninstall it and install an older version (from the sixth of January) and block updates.

In the working version of Nightly, could you set browser.enableAboutThirdParty to true on about:config, go to about:support, and see what are listed in the "Third-Party Modules" section at the bottom of the about:support page? That section shows modules which are injected into Firefox by third-party applications. If the patch for Bug 1684532 causes a launching problem, there should be an injected module which does not work well with our security feature.

It says No third-party modules were loaded.

Ignore this comment.

(In reply to Florencia Di Ciocco from comment #14)

It says No third-party modules were loaded.

Was this with an older build of Nightly and default compatibility mode (Windows 10 or already in Windows 8 compatibility mode?

(In reply to Sebastian Hengst [:aryx] (needinfo on intermittent or backout) from comment #17)

(In reply to Florencia Di Ciocco from comment #14)

It says No third-party modules were loaded.

Was this with an older build of Nightly and default compatibility mode (Windows 10 or already in Windows 8 compatibility mode?

It was with the version from the sixth of January and without the compatibility mode. If it matters, I'm from Argentina using the en-us build.

With the compatibility mode and 86.0a1 (2021-01-10) (64-bit) I see this

(In reply to Florencia Di Ciocco from comment #19)

Created attachment 9197483 [details]
third-party modules

With the compatibility mode and 86.0a1 (2021-01-10) (64-bit) I see this

Thank you for sharing the screenshot! Seems like there are three applications FireEye, Morphisec, and Cisco who inject their module into Firefox. Is it possible that you can temporarily disable each of them (or add Firefox.exe to an exception list if exists) and figure out which one (or which combination) prevents Firefox from launching?

(In reply to Bogdan Maris [:bogdan_maris], Release Desktop QA from comment #9)

Someone from Softvision IT checked the logs of Cisco AMP on my PC and found this:

An attack was prevented in kernel32.dll at base address 0x00007FF94CEE0000 inside the firefox.exe process.
File signed by Mozilla Corporation with certificate serial 0 from DigiCert SHA2 Assured ID Code Signing CA.
Expires NaN:NaN:NaN, NaN 0NaN UTC. The certificate was warn trusted

They reached out to Cisco to analyze this.

Thank you for communication with Cisco. My guess from their log message is Cisco's AMP considered our behavior, blocking their injected module malicious, thus they quarantined Firefox.

Sorry that the screenshot is in Spanish. Based on your comment, I started with Cisco and that solved the problem. I set Cisco to "run in Windows 8 compatibility mode" and I was instantly able to run Nightly. Thing is, this way, my old profile doesn't work. I have to create a new one.

We are still waiting for an update from our contact in Cisco. (In reply to Toshihito Kikuchi [:toshi] from comment #20)

Thank you for communication with Cisco. My guess from their log message is Cisco's AMP considered our behavior, blocking their injected module malicious, thus they quarantined Firefox.

We are still waiting for an update from our contact in Cisco unfortunately. Will provide more info as soon as we get something from them.

I'd like to chime in and state that I'm running into the issue where Nightly will not start because Amp is seeing it as an attack.

This is the error message in Event Viewer:

"An exploit attempt was prevented in C:\Program Files\Firefox Nightly\firefox.exe process with PID 6532 and needed to be shut down."

Forgot to mention that this is on Windows 10 Build 19042.746, 64-bit.

Thank you for reporting the problem. Apparently Cisco AMP does not like us self-protecting our process from their injection. Could you use Windows 8 compatibility mode as a workaround, or add firefox.exe to the exclusion list? I contacted our own contact for Cisco on Thursday, but no response yet.

(In reply to Toshihito Kikuchi [:toshi] from comment #25)

Thank you for reporting the problem. Apparently Cisco AMP does not like us self-protecting our process from their injection. Could you use Windows 8 compatibility mode as a workaround, or add firefox.exe to the exclusion list? I contacted our own contact for Cisco on Thursday, but no response yet.

Unfortunately I don't have the rights to change the exclusion list since it's an Active Directory-joined machine.

I just enabled Win 8 compatibility mode and it now successfully launches.

I wanted to add that when I first turn on the computer, Nightly works fine. I can open it without a problem. Later on, when I close it I can't open it again unless I run the compatibility mode.

I leave you my about:support of the first run. I mention this in case it helps.

Toshi, shoud this bug be assigned to you?

Just as an FYI, we are still waiting for a reply from Cisco, unfortunately. Last message from them was that they reproduced the issue and looking into it.

Nils, glad to hear the compat mode mitigated the issue. Sorry for inconvenience.

(In reply to Bogdan Maris [:bogdan_maris], Release Desktop QA from comment #29)

Just as an FYI, we are still waiting for a reply from Cisco, unfortunately. Last message from them was that they reproduced the issue and looking into it.

That's a great news that they reproduced the issue. Let's wait for their feedback.

As I wrote in comment 8, our new self-protection behavior bug 1684532, which Cisco AMP seems to consider malicious, is currently limited to Nightly only. So this will not happen in Beta or Release version.

Interestingly enough, if I launch Nightly after turning on compatibility mode, I can then close Firefox and disable compatibility mode. Then it will launch just fine, until the next update to Nightly happens. Then I repeat the process again.

The last couple of days I've been able to launch Nightly without needing compatibility mode. I am using the latest AMP definitions.

(In reply to Nils from comment #32)

The last couple of days I've been able to launch Nightly without needing compatibility mode. I am using the latest AMP definitions.

Thank you for letting us know. That's interesting. Hopefully AMP has changed their behavior.

Could you change browser.enableAboutThirdParty to true on about:config, go to about:support, and see if any Cisco module is displayed in the "Third-Party Modules" section at the bottom of the about:support page? If no Cisco module is listed, they refrained from injection. If there is, please click the down-arrow button in the Occurrence column of a Cisco module's row and check the value of the Status column?

There is a Cisco module loaded:
damsicom64.dll 7.3.3.5 Cisco Systems, Inc.

Process Type & ID Thread Imagebase Address Process Uptime (ms) Loading Duration (ms) Status
browser.0x1bd4 (No value) 0x7ff9f9500000 1458 1.7052 Loaded

(In reply to Nils from comment #34)

There is a Cisco module loaded:
damsicom64.dll 7.3.3.5 Cisco Systems, Inc.

Process Type & ID Thread Imagebase Address Process Uptime (ms) Loading Duration (ms) Status
browser.0x1bd4 (No value) 0x7ff9f9500000 1458 1.7052 Loaded

Thanks! Given that the status is "Loaded", it seems that the latest version of AMP uses an injection technique which is not automatically prevented.

Actually, I think this could be something to do with FF and not Cisco. Yesterday I was trying to do a mozregression and when the bisection was drawing near to mid-January I started to get the error shown on the pic. After much stress, I realized it was related to this bug. So I set the mozregression on compatibility mode and I got rid of the error message.

Just now I tried to run FF nightly without compatibility mode, and it loaded. Yet if I try to run a single build of nightly from January the 17 (for example) I can't, I get an error. If I run the same single build but having mozregression on compatibility mode, mozregression runs without problem.

So, If Nightly can run, Cisco has not changed, why can't I run an old version without using compatibility mode? This is why I believe that this could have something to do with FF.

I did a mozregression to see what might have caused this behavior on FF and I got this result: https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=24bc78c0fdc52ca31b540f959d6e53cb61a67f82&tochange=658693414024ff3409e24b58a32336dab7719d23

Yes, this issue is related to bug 1684532 landed on Jan-14. With that patch, we introduced a new self-protection behavior to prevent a specific injection technique we don't support. Cisco AMP is one of them using that technique, and it seems they consider our self-protection malicious. That's why we're contacting Cisco to ask them to change their injecting behavior.

Oh, ok, thanks for the answer. I thought I was adding new info.

FYI, I did have to revert to compatibility mode this morning. Once I successfully launched, I then disabled compatibility mode and was able to re-launch the software. I did confirm the 3rd party Cisco module was still loaded.

Toshi, do you really think it deserves a S1 severity?
Reminders, it means
"(Catastrophic) Blocks development/testing, may impact more than 25% of users, causes data loss, likely dot release driver, and no workaround available "

This would likely block release if it wasn't nightly only, so I'd argue the severity is right.

OK, merci!

P1 -> We're actively tracking this but as it's not our bug there's nothing here for us to fix.

I'm trying to get this to get this to the right folks at Cisco. I'm surprised that AMP is tying to inject a DLL into FF. There any more info you can share on that ?

(In reply to Cullen Jennings from comment #45)

I'm trying to get this to get this to the right folks at Cisco. I'm surprised that AMP is tying to inject a DLL into FF. There any more info you can share on that ?

Thank you for your help! Here's a brief summary of the issue:

• AMP injects damsicom64.dll into Firefox by modifying the executable's Import Directory.
• We automatically tries to prevent the injection technique using Import Directory.
• AMP considers our prevention behavior malicious and prevents Firefox from launching.

We don't have a repro environment because we don't have AMP's license, but I think running the latest Nightly and AMP will reproduce the issue. This will not happen with Beta or Release version.

I did confirm this happens with a recent version of Amp, and FF nightly on Windows 10. Still trying to track down the right people to look at this.

We received an update from Cisco, and I quote:

Devs have identified the issue and the permanent fix for this will be included in AMP Connector version 7.4.1, which is tentatively set for release at the end of march. You will see mention of this in the AMP for Endpoints Release Notes once 7.4.1 is released.

Cisco shared a release candidate of the AMP Windows Connector v7.3.15 including a patch for this issue with us, and I've confirmed the latest Nightly started with that version normally.

Cisco confirmed their patch for this issue was released on March 31st as AMP Windows Connector v7.3.15. Here's their release notes. Resolving this issue as Fixed.

Verified that with the latest version of AMP 7.3.15 on multiple machines inside Softvision, Nightly can be successfully started without any workaround.