Closed Bug 1389889 Opened 7 years ago Closed 7 years ago

Nightly 57 causes Win10 to BSOD with "0xD1: DRIVER_IRQL_NOT_LESS_OR_EQUAL" due to Bitdefender firewall component

Categories

(Core :: General, defect, P2)

57 Branch
x86_64
Windows 10
defect

Tracking

()

RESOLVED INCOMPLETE
Tracking Status
firefox-esr52 --- wontfix
firefox55 --- wontfix
firefox56 --- wontfix
firefox57 + wontfix

People

(Reporter: ilikethepussyboss, Unassigned)

References

(Blocks 1 open bug)

Details

(Keywords: crash, nightly-community, regression, Whiteboard: [gfx-noted][AV:Bitdefender Internet Security])

User Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; .NET4.0C; .NET4.0E; .NET CLR 2.0.50727; .NET CLR 3.0.30729; .NET CLR 3.5.30729; rv:11.0) like Gecko

Steps to reproduce:

Try and run nightly.


Actual results:

I ran the latest version of Nightly on the latest version of Windows 10 and it crashed my computer. When it crashed my computer I was given the error code; "DRIVER IQL LESS OR EQUAL" I then re-installed nightly and the error continued. I then deleted all existing files of nightly on my computer (including the ones in regedit) But the error persisted.


Expected results:

Nightly should've opened correctly.
Please send a crash report and paste the ID here. https://support.mozilla.org/kb/mozillacrashreporter
Keywords: crash
https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/bug-check-0xd1--driver-irql-not-less-or-equal
Severity: normal → critical
OS: Unspecified → Windows 10
Hardware: Unspecified → x86_64
Summary: Nightly crashes every time I run it. → Nightly lead to "0xD1: DRIVER_IRQL_NOT_LESS_OR_EQUAL" BSOD on Windows 10 when every time I run it.
User Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:54.0) Gecko/20100101 Firefox/54.0

I have tested this issue on Windows 10 x64 with the latest Firefox release (54.0.1) and the latest Nightly (57.0a1-20170816100153) and haven't managed to reproduce it.
After launching the latest version of Nightly or Firefox release, the browser is successfully opened and no crash occurred or BSOD.

Can you also re-test the issue with a clean new Firefox profile? Also is the issue reproducible on the latest Firefox Release or Beta channels?
Flags: needinfo?(ilikethepussyboss)
I have seen this issue when running Nightly as well. It doesn't happen every time and since I'm a heavy Nightly user I'm not sure if it's tightly correlated. Will try to monitor more closely.
Any chance of getting the graphics section of about:support?  If it crashes on startup, will it run in the safe mode (hold down shift key and click on Firefox)?  about:support from either of those runs, before the crash, would be great.
Component: Untriaged → Graphics
Product: Firefox → Core
If this consistently happens, does changing value of layers.mlgpu.dev-enabled to false in about:config (and restarting) help?
Whiteboard: [gfx-noted]
One of the affected systems:

Description	NVIDIA GeForce GTX 1080 Ti
Vendor ID	0x10de
Device ID	0x1b06
Driver Version	22.21.13.8205
Driver Date	5-1-2017
Status: UNCONFIRMED → NEW
Ever confirmed: true
I've got an update here, NickN uninstalled an install of BitDefender and the problem seems to have gone away. So a couple of things:

1. ni to the reporter, what if any AV software do you have installed?

2. to Ryan, do we test with Bitdefender or other AV software? This can't be an isolated case. 

The thing that worries me here is that running Nightly crashed Nick's gaming machine, but release was fine. It was suggested to me today that beta-config builds of nightly might behave differently as well. Help! This seems complicated.
Flags: needinfo?(ryanvm)
I believe that most of the AV testing we do is on a case-by-case basis on request. I don't know offhand to what extent we smoketest the major vendors, but I'm redirecting to Andrei who would.
Flags: needinfo?(ryanvm) → needinfo?(andrei.vaida)
I've been battling this issue as well and I have indeed BitDefender installed. Sadly no related crash reports in either the pending nor submitted directories, otherwise I would provide one of those. I'm happy to help tracking this down, though I haven't encounter it yet in today's build (20170821100350).
Carl, is this something you might have some thoughts on? Wondering if the DLL interceptor might be involved here. I remember aklotz having a lot of issues with nVidia drivers at one point.
Flags: needinfo?(ccorcoran)
I would not be surprised if the culprit here is Bitdefender. AV and security apps like this install kernel-mode drivers and hook into Firefox. So if Firefox does something Bitdefender doesn't expect, it can BSOD, and will not generate a Firefox crash report. DLL interceptor may be a solution but we'll need to find a way to repro it.

Memory dump from the blue screen could help a lot.
Flags: needinfo?(ccorcoran)
Here's a "Small memory dump (256 KB)" from my latest blue screen:
https://drive.google.com/file/d/0Bw6iUfeeZC_uaGYyNklUYkNtbTQ/view?usp=sharing

I also have what Windows calls an "Active memory dump" of the same crash, which I can upload somewhere too if needed (almost 5 GB though).

Peculiar enough Nightly never manages to start again after the blue screen. I can see the process pop up in Task Manager for a second or two before it disappears. Reinstalling it solves the problem, but the happiness only lasts for a couple of seconds before the blue screen strikes again.
Jonas, thank you, this file does confirm almost certainly that Bitdefender's firewall component is the culprit. Feel free to unshare the dump file (there's always the possibility it contains sensitive info so better be on the safe side).

It shows the following crashing stack trace:

> 0: kd> k
>  # Child-SP          RetAddr           Call Site
> 00 ffffbf80`1b7a64f8 fffff803`85e008a9 nt!KeBugCheckEx
> 01 ffffbf80`1b7a6500 fffff803`85dfee7d nt!KiBugCheckDispatch+0x69
> 02 ffffbf80`1b7a6640 fffff803`85cd9c4f nt!KiPageFault+0x23d
> 03 ffffbf80`1b7a67d0 fffff802`3f5acb52 nt!KeAcquireInStackQueuedSpinLock+0x2f
> 04 ffffbf80`1b7a6800 fffff802`3f5ac83e afd!AfdProcessBufferSend+0x42
> 05 ffffbf80`1b7a6890 fffff802`3eb4c29d afd!AfdTLBufferedSendComplete+0x5e
> 06 ffffbf80`1b7a68f0 fffff802`3f97b31c tcpip!TcpInspectCompleteTcbSend+0xb9
> 07 ffffbf80`1b7a6970 fffff802`3f9517d5 NETIO!memset+0x375c
> 08 ffffbf80`1b7a69b0 fffff802`3ecd19c1 NETIO!NetioDereferenceNetBufferList+0xa5
> 09 ffffbf80`1b7a69f0 fffff802`3ecd1948 fwpkclnt!FwpsDereferenceNetBufferList0+0x61
> 0a ffffbf80`1b7a6a40 fffff802`3ecd36cc fwpkclnt!FwpsFreeCloneNetBufferList0+0x198
> 0b ffffbf80`1b7a6a80 fffff802`3fdd3bbe fwpkclnt!FwpsDiscardClonedStreamData0+0x1c
> 0c ffffbf80`1b7a6ab0 ffffab84`2e5c1c80 bdfwfpf+0x3bbe
> ....

Where that top frame is bdfwfpf, described as:

> fffff802`3fdd0000 fffff802`3fdfb000   bdfwfpf  T (no symbols)           
>     Loaded symbol image file: bdfwfpf.sys
>     Image path: \??\C:\Program Files\Common Files\Bitdefender\Bitdefender Firewall\bdfwfpf.sys
>     Image name: bdfwfpf.sys
>     Browse all global symbols  functions  data
>     Timestamp:        Wed May 31 14:19:51 2017 (592EB4E7)
>     CheckSum:         00029EBA
>     ImageSize:        0002B000
>     Translations:     0000.04b0 0000.04e4 0409.04b0 0409.04e4

Maybe this is enough to get a repro going.
See Also: → 1396394
Nick says this wasn't a problem with earlier versions of Firefox so it sounds like a regression we should try to get to the bottom of. Since we don't see crash reports we don't know how widespread this but if it's every bitdefender user, we could lose quite a few users to this when it ships. Do we have someone that could reach out to bitdefender and see if they can find anything on their end?
Keywords: regression
Priority: P3 → P1
Component: Graphics → General
Summary: Nightly lead to "0xD1: DRIVER_IRQL_NOT_LESS_OR_EQUAL" BSOD on Windows 10 when every time I run it. → Nightly 57 crashes Win10 with "0xD1: DRIVER_IRQL_NOT_LESS_OR_EQUAL" due to Bitdefender firewall component
I've re-classified this as core/general but please correct if you know of a better component to use for this bug. This seems entirely to be a catastrophic interaction between Nightly and Bitdefender's firewall component based on #c14, correct?
Adding abalan from Bitdefender, who's assisted on other bugs in the past.
(In reply to ilikethepussyboss from comment #0)
> User Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; .NET4.0C;
> .NET4.0E; .NET CLR 2.0.50727; .NET CLR 3.0.30729; .NET CLR 3.5.30729;
> rv:11.0) like Gecko
> 
> Steps to reproduce:
> 
> Try and run nightly.
> 
> 
> Actual results:
> 
> I ran the latest version of Nightly on the latest version of Windows 10 and
> it crashed my computer. When it crashed my computer I was given the error
> code; "DRIVER IQL LESS OR EQUAL" I then re-installed nightly and the error
> continued. I then deleted all existing files of nightly on my computer
> (including the ones in regedit) But the error persisted.
> 
> 
> Expected results:
> 
> Nightly should've opened correctly.

                                                            
                                                           ***UPDATE*** 

I have once again wiped my computer clean of any Nightly related files and reinstalled nightly the latest version of nightly but the error still purists even when I try to open nightly in safe mode. I also cant get a-hold of a crash-log because every time I try my computer crashes. 

My computer specs: 

Processor: Intel Core 15 7600k 
Ram: Corsair Vengeance - White (2x8GB) 
PSU: Corsair CS550M 
Memory: Samsung EVO 500GB SSD 
Mother Board: GeForce GTX 1050 TI Dual 4GB
Flags: needinfo?(ilikethepussyboss)
Here. 
Just to give everyone a heads-up: our guys have some issues reproducing this right now. They'll keep trying and, if needed, they'll join in on this thread and ask for some assistance. 

Cheers
Hi,
I've been trying to reproduce this issue and didn't succeed. I've made a access request for the dmp file but didn't get a answer yet so if someone could make it available it would be nice.
Also a question for the users that have this problem. Could you report the build number of the Bitdefender product? This should be available if the taskbar icon is right-clicked in the "About" section.

Thank you
@ Jeff, you removed "BSOD" from the bug summary. Is this bug a Firefox crash or a Windows BSOD?

@ People that have reproduced this bug: Does it affect both 32- and 64-bit Firefox? Are you all running Windows 10?
Flags: needinfo?(jgriffiths)
I saw it on x64 in windows 10 but did not try other variants.
(In reply to Ryan VanderMeulen [:RyanVM] from comment #9)
> I believe that most of the AV testing we do is on a case-by-case basis on
> request. I don't know offhand to what extent we smoketest the major vendors,
> but I'm redirecting to Andrei who would.

Apologies for the late reply. Indeed, we are only testing Antivirus products on request, there's no general coverage ensured by our teams. We don't have any Bitdefender licenses available to test this.
Flags: needinfo?(andrei.vaida)
I encountered this on Windows 10 (x64) running the 64-bit Nightly.

I refrained from using Nightly since my last comment up until earlier this week, and I have not had a single BSOD since. Not sure if this is due to Nightly/Bitdefender/Windows fixing something (un)related or if I've just been lucky, but I've gone from BSOD every time I launch Nightly to not a single one in 2-3 days.

I've also made the memory dump posted earlier public again, and if I do encounter another BSOD I'll post another memory dump as well as the version/build number of all the things™.
I was also affected by the BSOD-bug with my Windows 10 and the Bitdefender Firewall (I use the latest Bitdefender Internet Security 2018).

This error has also not occurred with me for a long time. Recently there was an update of Bitdefender's Internet Security 2018 - I think this fixed this problem. Nightly is working for me perfectly.
Flags: needinfo?(jgriffiths)
Summary: Nightly 57 crashes Win10 with "0xD1: DRIVER_IRQL_NOT_LESS_OR_EQUAL" due to Bitdefender firewall component → Nightly 57 causes Win10 to BSOD with "0xD1: DRIVER_IRQL_NOT_LESS_OR_EQUAL" due to Bitdefender firewall component
Could someone please respond to Liviu Mihailescu's comment @ https://bugzilla.mozilla.org/show_bug.cgi?id=1389889#c20 ?

Thanks,
(In reply to Liviu Mihailescu from comment #20)
> Hi,
> I've been trying to reproduce this issue and didn't succeed. I've made a
> access request for the dmp file but didn't get a answer yet so if someone
> could make it available it would be nice.
> Also a question for the users that have this problem. Could you report the
> build number of the Bitdefender product? This should be available if the
> taskbar icon is right-clicked in the "About" section.
> 
> Thank you

The build number from Bitdefender Total Security now is 22.0.10.141. I hope this helps.
Hello,
I have been testing this issue since Friday but to no avail. I tried reproducing the BSOD on Windows 10 x86 & x64 machines using different versions of Bitdefender with a serial number that was provided directly by them. The build numbers of the Bitdefender products are 21.0.25.92, 21.2.25.33 and 22.0.10.141. The BSOD does not reproduce on any of these builds no matter how I used the Nightly build. I've used a clean profile and a "dirty" one, different Nightly builds from different months and still nothing. 

@Michael, did you have any add-ons installed on Nightly, maybe some other 3rd party software opened while browsing? Could you tell us your graphics card model?
Flags: needinfo?(mtesch)
See Also: → 1392463
Track 57+ to closely monitor the issue.
Reporter, based on comment 29, are you still able to reproduce this issue?
Flags: needinfo?(ilikethepussyboss)
Priority: P1 → P2
I'm curious what product and version of Bitdefender caused this, waiting on initial reporter to see if they still experience the problem.

Known good product versions: Bitdefender products 21.0.25.92, 21.2.25.33 and 22.0.10.141.
Blocks: injecteject
Whiteboard: [gfx-noted] → [gfx-noted][AV:Bitdefender Internet Security]
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → INCOMPLETE
Flags: needinfo?(ilikethepussyboss)
You need to log in before you can comment on or make changes to this bug.