Sylvestre Ledru [:Sylvestre] Reporter
	Description • 8 years ago

+++ This bug was initially created as a clone of Bug #1291738 +++

+++ This bug was initially created as a clone of Bug #828184 +++

My initial patch was incorrect, we have twice the same dll in the blocklist. 
We should have only one

	(No longer employed by Mozilla) Aaron Klotz
	Comment 2 • 8 years ago
mozreview-review

Comment on attachment 8780426 [details]
Bug 1294650 - Block the version of websense crashing 48 & 49. Followup, we already had this dll

https://reviewboard.mozilla.org/r/71158/#review68808

	(No longer employed by Mozilla) Aaron Klotz
	Comment 3 • 8 years ago
mozreview-review

Comment on attachment 8780426 [details]
Bug 1294650 - Block the version of websense crashing 48 & 49. Followup, we already had this dll

https://reviewboard.mozilla.org/r/71160/#review68810

	Pulsebot
	Comment 4 • 8 years ago

Pushed by sledru@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/94d311a8474c
Block the version of websense crashing 48 & 49. Followup, we already had this dll r=aklotz

	Sylvestre Ledru [:Sylvestre] Reporter
	Comment 5 • 8 years ago

Comment on attachment 8780426 [details]
Bug 1294650 - Block the version of websense crashing 48 & 49. Followup, we already had this dll

Approval Request Comment
[Feature/regressing bug #]: Outside change
[User impact if declined]: Startup crash
[Describe test coverage new/current, TreeHerder]: I don't have a windows to test, the patch is trivial
[Risks and why]: We are dealing with startup crashes, we cannot make it worst...
[String/UUID change made/needed]: None

	Gerry Chang [:gchang]
	Comment 6 • 8 years ago

Comment on attachment 8780426 [details]
Bug 1294650 - Block the version of websense crashing 48 & 49. Followup, we already had this dll

This patch fixes the crash by blocking websense. Take it in all 48/49/50.

	Mark Banner (:standard8)
	Comment 7 • 8 years ago
bugherder uplift

https://hg.mozilla.org/releases/mozilla-aurora/rev/a609f6015009

	Mark Banner (:standard8)
	Comment 8 • 8 years ago
bugherder uplift

https://hg.mozilla.org/releases/mozilla-beta/rev/2b4a93c456ae

	Mark Banner (:standard8)
	Comment 9 • 8 years ago
bugherder uplift

https://hg.mozilla.org/releases/mozilla-release/rev/4cc7ecd49f95

	Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)
	Comment 10 • 8 years ago
bugherder

https://hg.mozilla.org/mozilla-central/rev/94d311a8474c

	[:philipp]
	Comment 11 • 8 years ago

the issue seems to still be ongoing according to the first incoming crash reports from 48.0.1 like https://crash-stats.mozilla.com/report/index/88d96bf4-87bb-40ec-83fd-be2ce2160818

	Sylvestre Ledru [:Sylvestre] Reporter
	Comment 12 • 8 years ago

Oh... This is the same version (7.6.818.1) as we were supposed to block in my commit.
Benjamin, Aaron, any ideas? Not sure what to do. We received more than 1000 crashes (installation) last week. I am afraid that we will get 4 to 5 more crashes if we turn on updates for 48.0.1

	(No longer employed by Mozilla) Aaron Klotz
	Comment 13 • 8 years ago

We'd have to debug an Fx installation with this DLL present to know for sure, but typically if this happens then the DLL is being injected before the blocklist is able to catch it.

	[:philipp]
	Comment 14 • 8 years ago

hi, are you aware about that issue (startup crashes with firefox 48 and websense - also see bug 1291738)?

	Sylvestre Ledru [:Sylvestre] Reporter
	Comment 15 • 8 years ago

Aaron, philipp mentioned this two comments:
https://bugzilla.mozilla.org/show_bug.cgi?id=1291738#c23 and https://bugzilla.mozilla.org/show_bug.cgi?id=1181091#c59
Would they be acceptable for you?

	(No longer employed by Mozilla) Aaron Klotz
	Comment 16 • 8 years ago

(In reply to Sylvestre Ledru [:sylvestre] from comment #15)
> Aaron, philipp mentioned this two comments:
> https://bugzilla.mozilla.org/show_bug.cgi?id=1291738#c23 and
> https://bugzilla.mozilla.org/show_bug.cgi?id=1181091#c59
> Would they be acceptable for you?

Dropping a dummy dll into our directory would work (assuming that they're not loading with an absolute path, which appears to be the case based on those comments), but it's not particularly sustainable as a fix. I doubt that we want to be dropping dummy files into our directory every time another DLL like this shows up.

	Sylvestre Ledru [:Sylvestre] Reporter
	Comment 17 • 8 years ago

Aaron, as a temporary workaround for a 48.0.2, would you sign off on this? (or implement it ;)

	Lawrence Mandel [:lmandel] (use needinfo)
	Comment 18 • 8 years ago

Instead of pushing this to everyone, can we check for the existance of websense and create this dummy dll dynamically? Perhaps we can use the webcompat system add-on to push this fix out to affected systems before re-enabling updates. In this way we can remove the hack once websense pushes a fix (assuming that they do).

	Jim Mathies [:jimm] Assignee
	Comment 19 • 8 years ago

(In reply to Lawrence Mandel [:lmandel] (use needinfo) from comment #18)
> Instead of pushing this to everyone, can we check for the existence of
> websense and create this dummy dll dynamically? Perhaps we can use the
> webcompat system add-on to push this fix out to affected systems before
> re-enabling updates. In this way we can remove the hack once websense pushes
> a fix (assuming that they do).

This would require write privs to the program files folder. Usually we have to elevate to write to this path so I don't think this is possible.

The dummy dll seems ok, although I worry about breaking websense, causing crashes on the system, etc.. That said though those two testers commented it worked so, maybe. I ping;d kev on irc, he's going to try to reach out to these guys. In the mean time we can put together and test a fix.

Are there testers tuned in here who have this software and can help test a fix?

	Jim Mathies [:jimm] Assignee
	Comment 20 • 8 years ago

Maybe I jumped to a conclusion - does the 'webcompat system add-on' have the ability to update files in program files?

	Jim Mathies [:jimm] Assignee
	Comment 21 • 8 years ago

Hey Sander, any chance you could zip up a copy of qipcap.dll and attach it here?

	Jim Mathies [:jimm] Assignee
	Comment 22 • 8 years ago

I have an older copy from bug 1181091. dmajor did some debugging of this in that bug as well. We passed on trying to work around it.

	Jim Mathies [:jimm] Assignee
	Comment 26 • 8 years ago

Not a huge fan of this dummy dll idea. it's a bit of a hack that can easily be worked around by the 3rd party vendor. Spoke to aklotz, he feels the same. It would be better to work with the vendor to prevent this from happening (and get a quick fix out if possible for 48).

I'll get a try build of this dll trick patch going so we can test to see if it works.

There may be some things we can do to improve our blocklisting on new os as well (bug 1291353) which I'll insert into the platform integration teams work queue.

	Sylvestre Ledru [:Sylvestre] Reporter
	Comment 27 • 8 years ago

> It would be better to work with the vendor to prevent this from happening (and get a quick fix out if 
> possible for 48).
I wish we could get in touch with someone who understands our pain. For now, we haven't been able to find this contact (as far as i know). Even if they fix the issue, it is going to take a few weeks before all their users are updated.

The dll workaround is crap, for sure. However, it is a good mitigation until we:
* work with websense
* have a better mechanism to block them

	Jim Mathies [:jimm] Assignee
	Comment 28 • 8 years ago

What's the time frame I'm working with here for 48.0.2?

	Sylvestre Ledru [:Sylvestre] Reporter
	Comment 29 • 8 years ago

It would be great to have 48.0.2 live middle of next week.
Having a patch for Monday would be great.

	Jim Mathies [:jimm] Assignee
	Comment 31 • 8 years ago

For testing - 

Nightly build:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=713b0aaa5f3ba46638357ad244775459e11a7f6f

Release (48):
https://treeherder.mozilla.org/#/jobs?repo=try&revision=40495b7a5aa766705d61f9712f079dc2affb6fe4

	Jim Mathies [:jimm] Assignee
	Comment 32 • 8 years ago

Sanders, would you mind taking these for a spin to see if the problem is resolved?

	Jim Mathies [:jimm] Assignee
	Comment 33 • 8 years ago

Ryan, can softvision help us here at all? Not sure if they have access to this software. If they do, maybe they could try to reproduce and then test these builds.

	Ryan VanderMeulen [:RyanVM]
	Comment 34 • 8 years ago

Florin, does SV have access to Websense?

	Sander Goudswaard
	Comment 35 • 8 years ago

I am unable to test for the next week due to holiday - besides we dropped the endpoint software in favor of an appliance (although I still have access to it). Will ping our partner to ask for collaboration.

	[SV Manager] Florin Mezei, QA (:FlorinMezei)
	Comment 36 • 8 years ago

(In reply to Ryan VanderMeulen [:RyanVM] from comment #34)
> Florin, does SV have access to Websense?

I see Mihai tried to verify the crash yesterday, without success (see https://bugzilla.mozilla.org/show_bug.cgi?id=1291738#c30). 

Mihai can you provide some details here, as to what you've tried, and how/if we have access to Websense?

	Mihai Boldan, Desktop QA [:mboldan]
	Comment 37 • 8 years ago

I was unable to replicate the startup crash because there's no Websense Endpoint client available for download anymore. AFAIK, other than using an actual Forcepoint license, there's no trial available for this software on the web.

I'm currently discussing options with our IT dept. but it might take a while. Is there someone with Websense installed that could confirm this fix?

	Sylvestre Ledru [:Sylvestre] Reporter
	Comment 38 • 8 years ago

Jim, can we land that at least on nightly (and maybe aurora) so that we get some testing coverage during the week end? Thanks

	u194118
	Comment 39 • 8 years ago

(In reply to Mihai Boldan, QA [:mboldan] from comment #37)
> I was unable to replicate the startup crash because there's no Websense
> Endpoint client available for download anymore. AFAIK, other than using an
> actual Forcepoint license, there's no trial available for this software on
> the web.
> 
> I'm currently discussing options with our IT dept. but it might take a
> while. Is there someone with Websense installed that could confirm this fix?

I have Websense installed, I can test it when it lands on nightly

	Jim Mathies [:jimm] Assignee
	Comment 40 • 8 years ago

Hey, can you test with this try build? 

https://archive.mozilla.org/pub/firefox/try-builds/jmathies@mozilla.com-40495b7a5aa766705d61f9712f079dc2affb6fe4/try-win32/

	Jim Mathies [:jimm] Assignee
	Comment 41 • 8 years ago

(In reply to Sylvestre Ledru [:sylvestre] from comment #38)
> Jim, can we land that at least on nightly (and maybe aurora) so that we get
> some testing coverage during the week end? Thanks

Nightly would be ok I guess. There's no point in uplifting to release and generating a point release without confirmation it addresses the issue though.

	(No longer employed by Mozilla) Aaron Klotz
	Comment 43 • 8 years ago

Comment on attachment 8783358 [details] [diff] [review]
patch

Review of attachment 8783358 [details] [diff] [review]:
-----------------------------------------------------------------

::: toolkit/library/dummydll/dummydll.cpp
@@ +8,5 @@
> +  HANDLE hinstDLL,
> +  DWORD dwReason,
> +  LPVOID lpvReserved
> +)
> +{

Nit: It might be nice to include a call to DisableThreadLibraryCalls here.

	Pulsebot
	Comment 44 • 8 years ago

Pushed by jmathies@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/d332de446548
Install shim 'qipcap' dlls into the Firefox folder to circumvent dll injection by the 3rd party Websense product. r=aklotz

	Jim Mathies [:jimm] Assignee
	Comment 45 • 8 years ago

One concern I thought of is with updates, if this DLL gets loaded by another process and held, it could trigger restarts during software updating. Our update mechanism should be able to deal with this but it would make updating more cumbersome.

	Jim Mathies [:jimm] Assignee
	Comment 46 • 8 years ago

(In reply to Mihai Boldan, QA [:mboldan] from comment #37)
> I was unable to replicate the startup crash because there's no Websense
> Endpoint client available for download anymore. AFAIK, other than using an
> actual Forcepoint license, there's no trial available for this software on
> the web.
> 
> I'm currently discussing options with our IT dept. but it might take a
> while. Is there someone with Websense installed that could confirm this fix?

FWIW, I looked all over the web for an orphaned installer of this websense product with no luck. The company was purchased on January 2016 and rebranded to Forcepoint. Apparently they don't provide free trials anymore.

	Sylvestre Ledru [:Sylvestre] Reporter
	Comment 47 • 8 years ago

ok, looks like we have to take into aurora & beta at least and see what happens... :/

	Sylvestre Ledru [:Sylvestre] Reporter
	Comment 48 • 8 years ago

Andrei, I know this is a hard and painful but could you help to reproduce the crash and see if this is fixed in the try builds? Thanks

	Andrei Vaida [:avaida]
	Comment 49 • 8 years ago

(In reply to Sylvestre Ledru [:sylvestre] from comment #48)
> Andrei, I know this is a hard and painful but could you help to reproduce
> the crash and see if this is fixed in the try builds? Thanks

Unfortunately, I can only confirm what Mihai (Comment 37) and Jim (Comment 46) said about the websense product availability: there's no way for us to test this, because we can't find an installer for it anywhere.

We'd gladly help out here if, perhaps, someone could provide a trial installer for it.

	u194118
	Comment 50 • 8 years ago

Is the fix included in the latest nightly so I can test it ?

	Sylvestre Ledru [:Sylvestre] Reporter
	Comment 51 • 8 years ago

jcleval, you can test with https://archive.mozilla.org/pub/firefox/try-builds/jmathies@mozilla.com-40495b7a5aa766705d61f9712f079dc2affb6fe4/try-win32/

Any help would be appreciated!

	Jim Mathies [:jimm] Assignee
	Comment 52 • 8 years ago

(In reply to Pulsebot from comment #44)
> Pushed by jmathies@mozilla.com:
> https://hg.mozilla.org/integration/mozilla-inbound/rev/d332de446548
> Install shim 'qipcap' dlls into the Firefox folder to circumvent dll
> injection by the 3rd party Websense product. r=aklotz

This didn't merge into today's build. I'll get that done and we'll trigger a new nightly build.

	Pulsebot
	Comment 53 • 8 years ago

Pushed by jmathies@mozilla.com:
https://hg.mozilla.org/mozilla-central/rev/bad612fd3ab0
Install shim 'qipcap' dlls into the Firefox folder to circumvent dll injection by the 3rd party Websense product. r=aklotz a=jimm

	Ryan VanderMeulen [:RyanVM]
	Comment 54 • 8 years ago
bugherder

https://hg.mozilla.org/mozilla-central/rev/d332de446548

	Sylvestre Ledru [:Sylvestre] Reporter
	Comment 55 • 8 years ago

I asked releng (rail, thanks!) to restart a build of nightly with this fix.

	Sylvestre Ledru [:Sylvestre] Reporter
	Comment 56 • 8 years ago

Jim, could you fill the uplift requests to all branches so that we are ready once it is verified. Thanks

	u194118
	Comment 57 • 8 years ago

I can confirm that last nightly is not crashing on the same pc on which Firefox 48.0.1 is.
Thanks for the fix.

	Jim Mathies [:jimm] Assignee
	Comment 58 • 8 years ago

patch that landed.

	Jim Mathies [:jimm] Assignee
	Comment 59 • 8 years ago

Comment on attachment 8784089 [details] [diff] [review]
patch

Approval Request Comment
[Feature/regressing bug #]:
Crash caused by 3rd party security suite code injection. Not our bug.
[User impact if declined]:
Crashy startup for some users with the security suite installed.
[Describe test coverage new/current, TreeHerder]:
work around is on nightly which was tested by a single user who confirmed the fix works.
[Risks and why]:
medium risk, patch adds a dummy dll to our release install. stand alone this shouldn't cause any problems, hard to say what will happen though since the dll gets loaded by a 3rd party product. we do have confirmation that it addresses the startup crash caused by the security suite.
[String/UUID change made/needed]:
none

	Sylvestre Ledru [:Sylvestre] Reporter
	Comment 60 • 8 years ago

Comment on attachment 8784089 [details] [diff] [review]
patch

Taking it to all branches to test + build 48.0.2

	Sylvestre Ledru [:Sylvestre] Reporter
	Comment 61 • 8 years ago

jcleval: many many thanks :)

	Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)
	Comment 62 • 8 years ago

https://hg.mozilla.org/releases/mozilla-aurora/rev/a72bfbdf5c9b
https://hg.mozilla.org/releases/mozilla-beta/rev/29dd7d227b7b
https://hg.mozilla.org/releases/mozilla-release/rev/d4af06710040

	Brad Lassey [:blassey] (use needinfo?)
	Comment 63 • 8 years ago

(In reply to jcleval from comment #57)
> I can confirm that last nightly is not crashing on the same pc on which
> Firefox 48.0.1 is.
> Thanks for the fix.

Was nightly crashing you before this fix landed? If you weren't testing Nightly prior to the fix landing, could you try installing http://ftp.mozilla.org/pub/firefox/nightly/2016/08/2016-08-22-03-04-24-mozilla-central/firefox-51.0a1.en-US.win32.installer.exe to see if it crashes?

Alternatively, could you test the candidate beta build with the fix? http://ftp.mozilla.org/pub/firefox/candidates/48.0.2-candidates/build1/win32/en-US/Firefox%20Setup%2048.0.2.exe

Thanks

	Sylvestre Ledru [:Sylvestre] Reporter
	Comment 64 • 8 years ago

We have only one crash for now. 
https://crash-stats.mozilla.com/report/index/b5322f8f-89ef-4785-9068-efbfe2160824

Not panicking but if we get more, we should disable updates.

	u194118
	Comment 65 • 8 years ago

Hi.
I wasn't using nightly before the fix.
However I just upgraded FF 48.0.1, which was crashing on this pc, to 48.0.2, and the installation went well, no startup crash.

	Jonathan Howard
	Comment 66 • 8 years ago

More crashes in 48.0.2. (Not mine, don't run the software.)

Would adding to nsPrefBranch::GetIntPref a check of _retval == 0x2 be an (unideal) avoidance to the crash?

	Sylvestre Ledru [:Sylvestre] Reporter
	Comment 67 • 8 years ago

With a bunch of crashes, I don't think we can say that we fixed it.

	Jim Mathies [:jimm] Assignee
	Comment 68 • 8 years ago

(In reply to Sylvestre Ledru [:sylvestre] from comment #67)
> With a bunch of crashes, I don't think we can say that we fixed it.

In the module list for these, I see QIPCAP.dll loaded from system32, as well as QIPOverlay.dll, both with proper version info so I don't think our shim dll is getting picked up.

	Jim Mathies [:jimm] Assignee
	Comment 69 • 8 years ago

debugging one of these dumps, I see the pointer for nsPrefBranch::GetIntPref's _retval pointing into the memory space of this 3rd party dll. so if this stack is valid then they either hooked our api or are calling it with invalid parameters.

It would be great if we could find an in-house test case/machine for this, then we could trace back to the landing that caused their software to fault.

	David Bolter [:davidb] (NeedInfo me for attention)
	Comment 70 • 8 years ago

(In reply to Jim Mathies [:jimm] from comment #19)
> I ping;d kev on irc, he's going to try to reach out to
> these guys.

Any update here? (Note they have a twitter account https://twitter.com/forcepointsec)

	Sylvestre Ledru [:Sylvestre] Reporter
	Comment 71 • 8 years ago

We have a contact with them (started)

	Loic
	Comment 72 • 8 years ago

User on SUMO reported crashes with Websense and FF48.0.2:
https://support.mozilla.org/fr/questions/1136342

	Sylvestre Ledru [:Sylvestre] Reporter
	Comment 73 • 8 years ago

They stated the following:
"This issue was fixed in our latest release (8.2.5) and we are checking whether this fix will be backported to older versions."

	Liz Henry (:lizzard) (relman/hg->git project)
	Comment 74 • 8 years ago

Jim, do you think we should back this change to add the dll, out from 49 and above, since it didn't work in 48.0.2?

	Jim Mathies [:jimm] Assignee
	Comment 75 • 8 years ago

(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #74)
> Jim, do you think we should back this change to add the dll, out from 49 and
> above, since it didn't work in 48.0.2?

Yes, this is covered by bug 1298157.

	Pulsebot
	Comment 76 • 8 years ago

Backout by kwierso@gmail.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/4f33d866ca80
Backed out changeset d332de446548

	Guilherme Lima
	Comment 77 • 8 years ago

Landed 24 hours ago https://hg.mozilla.org/mozilla-central/rev/4f33d866ca80

	Liz Henry (:lizzard) (relman/hg->git project)
	Comment 78 • 8 years ago

The dll is backed out from 49 in bug 1298157 now. I don't think this should block 49 anymore. 
Websense users who are still on pre-48 versions will not yet update to 49, but can still update to 47.0.1.
 
We may re-assess this in a few weeks and can change the update rules based on feedback from websense about their user update rate, since users with their latest update should get the fix that will prevent startup crash. I expect by 50 release we should be able to bring pre-48 users up to 50.

	Liz Henry (:lizzard) (relman/hg->git project)
	Comment 79 • 8 years ago

For followup, we will want to know,
- How many users are held back on the blocked websense versions (I think releng can tell us this)
- Has websense shipped its fixes and do we know the proportion/# of users on the fixed versions vs. the old ones?
- Once it hits some level we think it acceptable, to be determined, Then we ship something that allows pre-48 users update to 49. 
We may want to file that as a new bug or bugs.

	BugBot [:suhaib / :marco/ :calixte]
	Comment 80 • 8 years ago

Crash volume for signature 'PREF_GetIntPref':
 - nightly (version 52): 0 crashes from 2016-09-19.
 - aurora  (version 51): 7 crashes from 2016-09-19.
 - beta    (version 50): 61 crashes from 2016-09-20.
 - release (version 49): 1261 crashes from 2016-09-05.
 - esr     (version 45): 1 crash from 2016-06-01.

Crash volume on the last weeks (Week N is from 10-03 to 10-09):
            W. N-1  W. N-2
 - nightly       0       0
 - aurora        7       0
 - beta         52       9
 - release     410     851
 - esr           0       0

Affected platforms: Windows, Mac OS X

Crash rank on the last 7 days:
           Browser   Content     Plugin
 - nightly
 - aurora  #184
 - beta    #278
 - release #126      #468
 - esr

	BugBot [:suhaib / :marco/ :calixte]
	Comment 81 • 8 years ago

Crash volume for signature 'PREF_GetIntPref':
 - nightly (version 52): 3 crashes from 2016-09-19.
 - aurora  (version 51): 23 crashes from 2016-09-19.
 - beta    (version 50): 214 crashes from 2016-09-20.
 - release (version 49): 1663 crashes from 2016-09-05.
 - esr     (version 45): 1 crash from 2016-07-25.

Crash volume on the last weeks (Week N is from 10-17 to 10-23):
            W. N-1  W. N-2  W. N-3  W. N-4
 - nightly       0       3       0       0
 - aurora       12       3       7       0
 - beta         65      78      52       9
 - release     140     194     410     851
 - esr           0       0       0       0

Affected platforms: Windows, Mac OS X

Crash rank on the last 7 days:
           Browser   Content     Plugin
 - nightly
 - aurora  #91
 - beta    #590
 - release #391      #678
 - esr

	Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)
	Comment 82 • 8 years ago

Hi Jim, I would prefer to add the shim in 50 if that is show the 49.0.1+ dot releases were pushed out. What do you think?

	Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)
	Comment 83 • 8 years ago

I've asked Wes to help. See https://bugzilla.mozilla.org/show_bug.cgi?id=1304783#c4

	Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)
	Comment 84 • 8 years ago

Removing the blocking flag, still keeping it tracked for 50.

	Pulsebot
	Comment 85 • 7 years ago

Pushed by kwierso@gmail.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/bba5ddb15392
Install shim 'qipcap' dlls into the Firefox folder to circumvent dll injection by the 3rd party Websense product. r=aklotz a=ritu

	Ryan VanderMeulen [:RyanVM]
	Comment 86 • 7 years ago

Do we really need this on trunk given the changes that have landed there RE: DLL injection?

	Benjamin Smedberg
	Comment 87 • 7 years ago

Possibly yes. My code makes Firefox less easily hackable, but it's not going to be perfect.

	Ryan VanderMeulen [:RyanVM]
	Comment 88 • 7 years ago
bugherder

https://hg.mozilla.org/mozilla-central/rev/bba5ddb15392

	Jorg K (CEST = GMT+2)
	Comment 89 • 7 years ago

Ported to TB/IB/SM:
https://hg.mozilla.org/comm-central/rev/ccb3cee77ef2530de35ba83ee6a64f957271ccbb

	Alexandru Simonca [:asimonca], Release Desktop QA
	Comment 90 • 7 years ago

I have tested this issue on the latest Beta and Release channel builds on a Windows 10 (x32 & x64) machine. 
I have experienced crashes on all of the following scenarios:

WebSense Endpoint Package 8.2.2324 on Firefox 51.0.1 build 2 x64 - Windows 10 x64
WebSense Endpoint Package 8.2.2324 on Firefox 51.0.1 build 2 x32 - Windows 10 x64
WebSense Endpoint Package 8.2.2324 on Firefox 51.0.1 build 2 x32 - Windows 10 x32

WebSense Endpoint Package 8.2.2324 on Firefox 51.0.1 build 3 x64 - Windows 10 x64
WebSense Endpoint Package 8.2.2324 on Firefox 51.0.1 build 3 x32 - Windows 10 x64
WebSense Endpoint Package 8.2.2324 on Firefox 51.0.1 build 3 x32 - Windows 10 x32
- qipcap64.dll file is present in the installation folder of Fx 51.0.1 build 3 x64
- qipcap.dll file is present in the installation folder of Fx 51.0.1 build 3 x32

WebSense Endpoint Package 8.2.2324 on Firefox 52.0b1 build 2 x64 - Windows 10 x64
WebSense Endpoint Package 8.2.2324 on Firefox 52.0b1 build 2 x32 - Windows 10 x64
WebSense Endpoint Package 8.2.2324 on Firefox 52.0b1 build 2 x32 - Windows 10 x32

	[SV Manager] Florin Mezei, QA (:FlorinMezei)
	Comment 91 • 7 years ago

Related to the test results above, this is consistent with previous testing of Firefox versions with the shim dll applied (e.g. 48.0.2): 
- the shim dll fixes crashes like [1] and [2], reproduced with a very old endpoint version (v8.0.2076)
- the shim dll does not fix crashes like [@ PREF_GetIntPref ] - that we track here - with endpoint version v8.2.2324 (commercial version 8.2)
- some newer endpoint versions do NOT cause a crash, regardless of the presence of a shim dll or not (e.g. commercial version 8.2.5_2424 = 8.2.5.2424, or commercial version 8.3 = 8.3.2509)

So in fact the shim dll seems to have the same effect as in 48.0.2 for example, where it prevents startup crashes for users with some old Websense versions installed, but still cannot prevent the crash that we track here for some relatively newer Websense versions (e.g. 8.2).

More details on Websense testing until today can be found here - https://docs.google.com/spreadsheets/d/18n8NEimajNpnGRHdQsrrnyJ-Ufgvmpz-AXCUpVaFRMk/edit#gid=0.

[1] - [@ @0x0 | mozilla::ShutdownXPCOM] - https://crash-stats.mozilla.com/report/index/25e534a3-2e09-4348-ac5e-8e5952160914
[2] - [@ @0x0 | nsJSCID::GetService ] - https://crash-stats.mozilla.com/report/index/66153288-5453-46b1-bf86-1381f2160914

	f1enigma
	Comment 92 • 7 years ago

50.1.0 with Websense (ForcePoint) - 8.2 Build 2424 works great on 32bit Win 7, while the same configuration is not working on 51.0.1(In reply to Florin Mezei, QA (:FlorinMezei) from comment #91)
> Related to the test results above, this is consistent with previous testing
> of Firefox versions with the shim dll applied (e.g. 48.0.2): 
> - the shim dll fixes crashes like [1] and [2], reproduced with a very old
> endpoint version (v8.0.2076)
> - the shim dll does not fix crashes like [@ PREF_GetIntPref ] - that we
> track here - with endpoint version v8.2.2324 (commercial version 8.2)
> - some newer endpoint versions do NOT cause a crash, regardless of the
> presence of a shim dll or not (e.g. commercial version 8.2.5_2424 =
> 8.2.5.2424, or commercial version 8.3 = 8.3.2509)
> 
> So in fact the shim dll seems to have the same effect as in 48.0.2 for
> example, where it prevents startup crashes for users with some old Websense
> versions installed, but still cannot prevent the crash that we track here
> for some relatively newer Websense versions (e.g. 8.2).
> 
> More details on Websense testing until today can be found here -
> https://docs.google.com/spreadsheets/d/18n8NEimajNpnGRHdQsrrnyJ-Ufgvmpz-
> AXCUpVaFRMk/edit#gid=0.
> 
> [1] - [@ @0x0 | mozilla::ShutdownXPCOM] -
> https://crash-stats.mozilla.com/report/index/25e534a3-2e09-4348-ac5e-
> 8e5952160914
> [2] - [@ @0x0 | nsJSCID::GetService ] -
> https://crash-stats.mozilla.com/report/index/66153288-5453-46b1-bf86-
> 1381f2160914

I have WIN7 32bit with Forcepoint 8.2 build 2424. It works with 50.1.0 but not with 51.0.1

	Marco Castelluccio [:marco]
	Comment 93 • 7 years ago

(In reply to Florin Mezei, QA (:FlorinMezei) from comment #91)
> More details on Websense testing until today can be found here -
> https://docs.google.com/spreadsheets/d/18n8NEimajNpnGRHdQsrrnyJ-Ufgvmpz-
> AXCUpVaFRMk/edit#gid=0.

Do you have any details regarding compatibility with 51?

	[:philipp]
	Comment 94 • 7 years ago

(In reply to Ryan VanderMeulen [:RyanVM] from comment #86)
> Do we really need this on trunk given the changes that have landed there RE:
> DLL injection?

(In reply to Benjamin Smedberg [:bsmedberg] from comment #87)
> Possibly yes. My code makes Firefox less easily hackable, but it's not going
> to be perfect.

the good news is, that the crashes with involvement of the websense module really seem to stop in 53: https://crash-stats.mozilla.com/search/?proto_signature=~qipcap.dll&release_channel=aurora&date=%3E%3D2016-10-01T00%3A00%3A00.000Z&_sort=-date&_facets=signature&_facets=version&_facets=build_id&_facets=platform_pretty_version&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-version :))

	Ryan VanderMeulen [:RyanVM]
	Comment 95 • 7 years ago

Is there anything actionable left to do in this bug for 52/53? The shim DLL was landed across all branches in bug 1304783. And AFAICT, any future mitigations we land on our end aren't likely to be upliftable.

	Marco Castelluccio [:marco]
	Comment 96 • 7 years ago

(In reply to Ryan VanderMeulen [:RyanVM] from comment #95)
> Is there anything actionable left to do in this bug for 52/53? The shim DLL
> was landed across all branches in bug 1304783. And AFAICT, any future
> mitigations we land on our end aren't likely to be upliftable.

There is bug 1329692, we are protected from the websense crashes in the version after the version that will contain those changes.

	Sylvestre Ledru [:Sylvestre] Reporter
	Comment 97 • 7 years ago

I think we are as good as we can be here :/

	Ryan VanderMeulen [:RyanVM]
	Comment 98 • 7 years ago

I'm setting the tracking flags to fixed to get this specific bug off the radar in that case. We're obviously still tracking ongoing Websense issues via bug 1305847 and its dependencies, however.