Closed Bug 828632 Opened 12 years ago Closed 11 years ago

running proxy config code off the main thread on OS X causes memory leak, syslogd load

Categories

(Core :: Networking, defect)

18 Branch
All
macOS
defect
Not set
normal

Tracking

()

VERIFIED FIXED
mozilla21
Tracking Status
firefox18 + verified
firefox19 + verified
firefox20 + fixed
firefox21 --- fixed
b2g18 --- affected
b2g18-v1.0.1 --- affected

People

(Reporter: s.t.smith, Assigned: mcmanus)

References

Details

(Keywords: regression)

Attachments

(1 file)

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/534.52.7 (KHTML, like Gecko) Version/5.1.2 Safari/534.52.7

Steps to reproduce:

New upgrade to FF 18.0 on OS X Lion 10.7.2.

System Preferences>Network>Advanced...>Proxies>Automatic Proxy Configuration: file://localhost/Library/Internet Plug-Ins/proxy.pac

Firefox>Preferences>Advanced>Network>Settings...>Use system proxy settings

As this bug already has a RESOLVED status, I'll start a new ticket.


Actual results:

FF hangs, system log fills at huge rate with messages that look like:

org.mozilla.firefox objc  Object class __NSCFData autoreleased with no pool in place - just leaking - break on objc_autoreleaseNoPool() to debug

syslogd cant's keep up with the error load and the Console app pizza's until I tell FF to stop using my system proxy settings, which use a local proxy.pac file.



Expected results:

No memory leak, no syslogd errors when FF is set to use system proxy settings, which are set to use a local proxy.pac file.
Hardware: x86 → x86_64
Component: Untriaged → Networking
Product: Firefox → Core
Status: UNCONFIRMED → NEW
Ever confirmed: true
Any ideas Patrick? Since this seems to be fixed on m-c, perhaps this is a patch that just didn't get approved for the FF18 branch?
Cc'ing Asa to help expedite through rapid release.

So is this a dup too? What's the bug with the fix-patch?

/be
(In reply to Josh Aas (Mozilla Corporation) from comment #3)
> Any ideas Patrick? Since this seems to be fixed on m-c, perhaps this is a
> patch that just didn't get approved for the FF18 branch?

This is in my queue to look at.

I'm not 100% convinced this is fixed in nightly - would be worth testing that. If that's true its almost certainly for the lack of the same patches that weren't approved for 18 that are causing the problem in 828202. (which I don't know off the top of my head, but I can figure out.)

If you've got a mac handy can you confirm the 18/nightly (or 19) behavior? The only one I have is old enough to be a real pain to work with.
Need automated test coverage.

/be
(In reply to Brendan Eich [:brendan] from comment #4)
> Cc'ing Asa to help expedite through rapid release.
> 
> So is this a dup too? What's the bug with the fix-patch?
> 

If it is a dup of 828202 then the patches in question are 815783 and 819902
If you can reproduce this bug I'd appreciate you testing another build. I have not yet reproduced myself (still working on it), but there are a couple possible related patches that I have included in a test build (which is still compiling)

Builds will appear here based on FF18 with the prospective fixes included

http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/mcmanus@ducksong.com-4bd823c0f036

The builds could take a while (anywhere from 30 mins to a few hours) to appear.
I think there are two issues here.. one is a dup of 828202 - if you hit that (depends on your pac uri) then you get a crazy spin.. this is already fixed >= ff19. the other issue is a leak/syslog that seems to come from running the os x system interaction stuff off the main thread (I'm having trouble quantifying its size).. if you're not getting the spin then this log is obviously happening a lot less. That problem exists on 18->21.

to fix the latter, fortunately we've already got a mechanism to ping/pong this stuff between threads so it runs on the main thread. (linux uses it). so initial tests indicate this can be fixed just by flipping a bool.

I'll use this bug for the leak/log issue.
STS can you try the build below and report back if that resolves the problem for you?

http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/mcmanus@ducksong.com-2102aa36db35/try-macosx64/
Flags: needinfo?(s.t.smith)
Attached patch patch 0Splinter Review
so 769764 moved the code that interacts with the system proxy settings off the main thread.. that code invariably involves os specific calls which are kind of opaque and they could block (or be changed to block) as they have been observed to do on windows.

however, executing that code on the pac thread on OSX seems to result in a syslog (and maybe a memleak if the log is to be believed) which can pile up at a high rate of speed.

fortunately there is already a mechanism for running this stuff on the main thread (linux glib requires it) so that can easily be fixed - it fixes the syslog about the leak for me. The only risk is that the system call could block but I haven't observed OS X to ever do that.. and if it were to do so that would still be a smaller problem imo.

given the osx nature of this I think josh is probably the best r?
Attachment #701144 - Flags: review?(joshmoz)
Attachment #701144 - Flags: review?(joshmoz) → review+
Assignee: nobody → mcmanus
Blocks: 769764
Keywords: regression
Hardware: x86_64 → All
Summary: System Proxy Configuration Causes Memory Leak, syslogd load → running proxy config code off the main thread on OS X causes memory leak, syslogd load
(In reply to Patrick McManus [:mcmanus] from comment #10)
> STS can you try the build below and report back if that resolves the problem
> for you?
> 
> http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/mcmanus@ducksong.
> com-2102aa36db35/try-macosx64/

Thanks -- that build works for me!
Flags: needinfo?(s.t.smith)
Comment on attachment 701144 [details] [diff] [review]
patch 0

[Approval Request Comment]
Bug caused by (feature/regressing bug #): 769764
User impact if declined: users with proxies defined on OSX via System Configuration will see a serious CPU tax and maybe a severe memory leak. This involves repeated syslog warnings which will slow the system.
Testing completed (on m-c, etc.): on m-i, and reporter confirms fix in comment 13
Risk to taking this patch (and alternatives if risky): The patch is simple (1 line) and while it does change the runtime logic considerably by using a different data flow that flow is already used in linux.
String or UUID changes made by this patch: none
Attachment #701144 - Flags: approval-mozilla-release?
Attachment #701144 - Flags: approval-mozilla-beta?
Attachment #701144 - Flags: approval-mozilla-aurora?
Thanks STS!
https://hg.mozilla.org/mozilla-central/rev/3890eb79af81
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla21
Comment on attachment 701144 [details] [diff] [review]
patch 0

Please land on Aurora/Beta asap.
Attachment #701144 - Flags: approval-mozilla-beta?
Attachment #701144 - Flags: approval-mozilla-beta+
Attachment #701144 - Flags: approval-mozilla-aurora?
Attachment #701144 - Flags: approval-mozilla-aurora+
Attachment #701144 - Flags: approval-mozilla-release? → approval-mozilla-release+
Keywords: verifyme
Did some basic verification of this one:

Setup:
-- OS X 10.8
-- Squid proxy server running locally
-- Fx using System Settings
-- Mac Sys Prefs set to file://localhost/Users/steveworkman/proxySquid.pac

Fx18:
-- Problem reproduced: Memory leak messages observed in syslog with Squid stopped (2 errors) and started (~20 errors trying google.com).

Fx 18.0.1:
-- No memory leak messages in syslog with Squid stopped or started.
(In reply to Steve Workman [:sworkman] from comment #20)
> Did some basic verification of this one:
> 
> Setup:
> -- OS X 10.8
> -- Squid proxy server running locally
> -- Fx using System Settings
> -- Mac Sys Prefs set to file://localhost/Users/steveworkman/proxySquid.pac
> 
> Fx18:
> -- Problem reproduced: Memory leak messages observed in syslog with Squid
> stopped (2 errors) and started (~20 errors trying google.com).
> 
> Fx 18.0.1:
> -- No memory leak messages in syslog with Squid stopped or started.

FWIW, I'm also running a squid proxy when I see this bug.
verified fixed running 18.0.1  on OS X 10.6.8

No memory leak messages in syslog  with pac file
Status: RESOLVED → VERIFIED
Keywords: verifyme
Verified as fixed on Firefox 19 beta 2 - no memory leak messages in syslog after configuring a system proxy using a file:// pac file.

Verified on Mac OS X 10.7.5 and on Mac OS X 10.8.2:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:19.0) Gecko/20100101 Firefox/19.0 Build ID: 20130116072953
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:19.0) Gecko/20100101 Firefox/19.0 Build ID: 20130116072953
QA Contact: simona.marcu
This issue is also verified on Firefox 19 beta 6 (20130212082553), on the next machines:

1) Ubuntu 12.04 64-bit
2) Windows 7 64-bit
3) Windows 8 32-bit
4) Mac OSX 10.6.8
Batch edit: Bugs marked status-b2g18: affected after 2/13 branching of v1.0.1 are now also status-b2g18-v1.0.1: affected
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: