Closed Bug 1428239 Opened 6 years ago Closed 6 years ago

Extreme memory leak, growing processes

Categories

(Firefox :: General, defect, P3)

57 Branch
defect

Tracking

()

RESOLVED WORKSFORME

People

(Reporter: jbullis, Unassigned)

References

Details

(Whiteboard: [MemShrink:P3])

Attachments

(7 files)

User Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0
Build ID: 20180103231032

Steps to reproduce:

Leave a single Firefox window up with about 15 taps open.  It will max the physical memory in a couple of hours. 


Actual results:

System slowed to a crawl, it was virtually unresponsive.    

Took 3 minutes to load Task Manager.  There were at least 20 Firefox processes running and about half had 500 MB or more of memory allocated to them.  Firefox was set up for 7 processes, but it greatly exceeded the process limit.  All 16 GB of ram were used and the system was paging like crazy.    

Took about 8 minutes to delete all the Firefox processes and recover the memory.  Once Firefox was totally down, the system was running on 1.7 GB of memory was no paging.

System was too slow to get diagnostic dumps or other information.  Will attempt to do it later and update the ticket.


Expected results:

Process count to be limited to 7 that it was configured for and no memory leaks.
Apparently Windows 7 (after how many years) doesn't totally handle low memory situations.  There were a couple of problems with Windows, had to reboot.
There is a single window open with the following URLs open in tabs:

http://www.cnn.com/?refresh=1
https://www.msn.com/
https://www.msn.com/en-us/health/healthtrending/avoid-romaine-lettuce-for-now-consumer-reports-says/ar-BBHQrDP?li=BBnbfcL
https://www.msn.com/en-us/health/nutrition/15-health-foods-youre-better-off-avoiding/ss-BBH7wiq?li=BBnb7Kz#image=11
https://www.msn.com/en-us/foodanddrink/foodculture/100-foods-to-try-before-you-die/ss-BBH5ult?li=BBnb7Kz
https://www.rottentomatoes.com/search/?search=to%20the%20bone
https://www.welderseries.com/-p49762034
https://www.thrillist.com/entertainment/nation/best-tv-shows-2017-to-watch
https://www.thrillist.com/entertainment/nation/best-movies-of-2017
https://www.thrillist.com/entertainment/nation/best-albums-2017-new-music-releases
https://www.thrillist.com/entertainment/phoenix/things-to-do-in-phoenix-bucket-list
https://www.thrillist.com/travel/nation/best-places-to-visit-in-the-philippines
https://www.facebook.com/groups/corvairracers/
https://www.thrillist.com/entertainment/nation/whats-leaving-netflix-movies-shows
https://www.supercall.com/news/canada-wine-beer-train?utm_campaign=20171226%3A+Phoenix%3A+Thrillist+Daily+12%2F26%2F2017&utm_medium=email&utm_source=campaignmonitor&utm_content=body7
https://www.thrillist.com/entertainment/nation/best-songs-spotify
https://www.thrillist.com/entertainment/nation/new-on-netflix-movies-tv-shows-originals
https://www.thrillist.com/entertainment/nation/netflix-movies-tv-shows-coming-in-2018
https://www.thrillist.com/entertainment/nation/most-watched-viral-youtube-videos-2017
https://www.thrillist.com/entertainment/nation/best-netflix-tv-shows-original-series
https://www.thrillist.com/entertainment/nation/funniest-movie-sex-scenes
http://openloadmovies.tv/?s=strange+relations
https://dvd.netflix.com/Subscribe/EnableDvd/Address
Whiteboard: [MemShrink]
Noting the attachment you added and comment 2, I believe 10 Firefox processes do seem around the right number. When you say 7 processes, you would refer to content processes (see [1]). If I'm not mistaking, the remaining 3 processes would be: main process, quantum compositor and gecko [2].

I'm a bit baffled about the 20 processes you are talking about in comment 0, so could you please confirm that the number of processes hit 20?
Also, related to the memory consumption, would you be kind enough to check what extensions you are running and eventually list them? It might be that there is an extension or a combination of extensions + environment that is causing this, so it would be useful if you would add info related to the firewalls and antiviruses you are using.

A good test would be to check what the memory consumption is if you use a clean profile with the same pages opened.


[1] set the dom.ipc.processCount = 7 ( or https://support.mozilla.org/en-US/kb/performance-settings), which means 7 content processes. 
[2] a rather technical explanation of what's what and how it works in Firefox Quantum : https://hacks.mozilla.org/2017/11/entering-the-quantum-era-how-firefox-got-fast-again-and-where-its-going-to-get-faster/
Flags: needinfo?(jbullis)
There were definitely AT LEAST 20 firefox process active at one time.  Most of which had about 500 MB or memory allocated to them with some holding more memory.  Since that time, I have not found more than 10 processes.  Unfortunately, doing a single command at that point when there were 20+ processes and no memory took minutes, so even getting a screen shot of the task manager would have taken an hour or more at that point.  Just ending Firefox with task manager took 8 minutes and another 3 or 4 for Windows to reclaim all the memory.  That problem occurred while running version 57.0.3.  When I reloaded Firefox, it installed 57.0.4.  A review of the change log didn't indicate a fix for that problem, so I reported it.

The memory usage was a major problem for a couple of days.  Had to kill and restart Firefox several times a day to prevent it for disabling the computer.  The past couple of days, I've seen modest growth with the same pages active for hours at a time, but nothing like I experienced over the previous two days.  Of course, the URL mix has changed, so that may have something to do with it.  I suspect the issue is in some rarely use code such as an error routine.

The add-ons I running are as follows:

Add Block Plus
Behind the Overlay Revival
Forecast Fox (fix version)
Ghostery
Google Search Link Fix
Honey
Popup Blocker (Strict)  

I have not seen Firefox update any add-ons over this period,so very likely, these are the same add-ons that were running at the time.

I'm using Firefox Security Essentials with its firewall.  Most firewall functions are actually handled by my external router.

I will try a clean profile when I get a chance.
Most of the firewall is handled by the external router.
Flags: needinfo?(jbullis)
Microsoft Security Essentials not Firefox Security Essentials
Firefox today is running with 9 processes (configured for 7).  As I understand it, should be 10 (7 configured + 3 others).  Not running the normal 10 for some reason.  Two windows open with 6 tabs, between the 9 processes, using 6.5 GB of ram.
I first noticed the problem with ver. 57.0.3.
If you aren't the original reporter, please file a new bug.
The memory leak still exists in version 58.0.  Had a couple of process with over 1.5 GB of ram allocated to each of them with several others around 500 MB.  System was slowing due to paging.
Can you test with a standard configuration (4 content processes)? Also are you able to reproduce the issue in safe mode? It's possible one of your extensions or settings are causing issues.
Flags: needinfo?(jbullis)
Have set up Firefox for 4 processes and disabled all add on.  Will report back when I know what's happening.  May be a couple of days.  Firefox sometimes runs a couple of days before the problem shows up.  Sometimes happens 2 or 3 times a day.  When I finish that test, I will try it in safe mode with 4 processes and no add ons active.
Flags: needinfo?(jbullis)
This test was done with the standard 4 processes and all add-ons disabled.  Firefox at the time was using one window with approximately 40 tabs open.  The tabs were loaded via a session recovery and each tab was visited to make sure it populated completely.  Then one snap shot was taken of the Windows Task Manager.  A second snap shot was taken 8 hours later.  Three was no user input during this test.  Two of the windows were news URLs which update about every 30 seconds.  Perhaps 10% of the stories changed over the 8 hours.  

Watching memory usage via Window Task Manager showed that a considerable amount of activity was occurring within four of the processes each second per the Task Manager.  I'm not sure what Firefox is doing with little to no input occurring.

During this time Firefox behaved itself fairly well during this time with only modest growth in memory use (2.249GB to 3.234GB).  From experience, the memory use growth rate can vary wildly this being a period of modest growth.  There were times in the past where memory use grew by 4GB in 2 hours.

As far as extensions possibly causing the problem, it is possible that one of the extensions is adding to the problem, but for this run, they were all disabled, so Firefox core code definitely has memory leaks. 

This test was done with Firefox Quantum version 58.0.2(64-bit).  Firefox indicated that this was the latest version.

I will upload the two snap shots of the Windows Task Manager.
Snap shot start of test 4 processes no add-ons
Windows task manager snap shot 8 hours into the test
Whiteboard: [MemShrink] → [MemShrink:P3]
Until we know more and in order to gain more traction, I am moving this bug to ::General.

I haven't run a long test on Win7 x32 yet (20-30' or so) but it did give me the same impression that the memory is not managed quite properly. I am using quite heavily FF on my home computer(Win10), which is powered on 24/7 and as far as I remember, and I haven't noticed such problem, so we might be looking at a problem that might be Win7 specific. 
Adding a NI for myself as a reminder to followup with a longer win7 test.
Component: Untriaged → General
Flags: needinfo?(adrian.florinescu)
I have had this problem occur on my Windows 10 laptop also.  Definitely not just a Windows 7 problem.  Don't use the lap top much, so obviously don't see the problem as often.
Sorry it took me a while to get back to this bug, although this is such an important matter.
I did a few tests on Windows 10 x64, Ubuntu 16.04 and Windows 7 x 32, with the following details:
-FF 59.0.1 32b for windows 7 x32
-FF 59.0.1 64b for Windows 10 and Ubuntu 16.04
-test run of 2-3hrs with little to no interaction
-20 tabs with reddit, facebook, youtube, wikipedia (note: I didn't want to use any google results pages for this test due to bug 1218169)

On all OS'es and test runs, while I did have at some points some spikes in memory and CPU usage, they leveled after a while and I just couldn't see any extreme memory management issues that would make the OS memory spiral out of control. All the tests took in account ~3hrs of usage with little to no input after the initial tabs open/load.

Since obviously the above doesn't push this bug forward, could you perhaps try to reproduce the problem, while taking an about:memory snapshot at a half hour interval while the memory starts spiraling out of control? Jerry, Arseny would you please try to assist in moving this issue further? 

Steps:
1. Using a new profile, open Firefox 59.0.1 or latest release.
2. Open a tab with about:memory
3. Open 20 tabs, loading the content for each.
4. Each 1/2 hrs in the about:memory click on Measure and save, saving a memory report.
Flags: needinfo?(jbullis)
Flags: needinfo?(arseniy.zakharov)
Flags: needinfo?(adrian.florinescu)
I haven't had a problem on my Windows 7 machine in the past 2 weeks.  However, the problem has now showed up on my Windows 10 laptop.  I concur with bht237, it does look like it might be tied to certain sites or pages or perhaps some error process.   It is definitely not a main code path issue.  Closing the tabs doesn't free memory, but re-starting Firefox does.  Definitely a memory leak.
Flags: needinfo?(jbullis)
Status: UNCONFIRMED → NEW
Ever confirmed: true
I usually run Ghostery and another ad blocker, can't tolerate the junk ads either.  Just installed the new Firefox and it dumped virtually all the add-ons.  Interestingly enough, most are available to download from the Firefox site.  Had to re-install virtually all the add-ons.  Not sure why Firefox deleted them instead of updating them. 

Will say that the memory leak problem varies.  On one day, I have to kill Firefox on the desktop 7 or 8 times, then Firefox runs OK with a slight memory leak problem for the next week.  Right now, the 64 bit Firefox on my Windows 7 system hasn't has had a minor memory leak, can run for 3 to 5 days without restarting it.

I am having to reset the Windows 10 64 bit Firefox on my lap top 3 or 4 times per day at the moment.  I typically use each system for different things.  Trying to figure out if it site related.

Hopefully, the leak you found is the problem, but from history with Firefox, I suspect there is more than one leak.  It has not been a strong point of Firefox, but in the past the leaks were slow.

Will try playing with uBlock Origin and check zerohedge.com.  Will also try with and without Ghostery active.

Nice catch bht237!!!!!!
The performance on zerohedge.com is really bad. It doesn't seem related to the original issue, though. I'll file a separate issue for it.
I filed bug 1428239 for zerohedge.com.
If you are not the original reporter, please file a new bug, or comment in bug 1428239 if you want to discuss the specific zero hedge.com issue. You may be experiencing a different issue than the original reporter, and having a bunch of different issues jumbled up together in a single bug makes it very hard to figure out what is going on. Feel free to needinfo me if you do file such a bug.
When you start to experience the high memory usage, but before the memory usage is so high that you can't use Firefox at all, please go to about:memory and go to "measure and save" to get a memory report and upload it to this bug. Note that this report will contain all of the URLs you have open, so please check the "anonymize" box if you don't want them publicly visible in the bug. The memory report is the first step towards diagnosing a memory problem.
Flags: needinfo?(jbullis)
Massive memory use, see saved report as requested.
Flags: needinfo?(jbullis)
I don't need memory reports for zero hedge, thanks. I can reproduce that myself. I was wondering what the memory was for the original reporter on this bug. I don't think they had zero hedge loaded.
Flags: needinfo?(jbullis)
I think I found yet another site that causes memory leak (with uBlock Origin on this time).

https://opensource.com/article/17/8/reasons-i-come-back-gnome (not sure if it's a particular page or the site in general).

See if you could reproduce it and please let me know if you need additional info like about:memory.
Attached file memory report and save
I instance of Firefox running about 18 tabs.  Has been up for a few days.  Memory appears to be growing slowly.  This is on my Windows 7 machine.  

Windows 10 laptop at present having more problems, it only has 8GB of memory, so by the time I catch it, the system is usually too far gone.  Having to reset Firefox on the Windows 10 machine once in the morning and once in the evening to use.  Will continue to try and catch Windows 10 which is experiencing a greater problem at the moment.
Flags: needinfo?(jbullis)
This is pretty odd:

> ├────274.58 MB (24.19%) -- dom
> │    ├──274.17 MB (24.16%) ── structured-clone-holder [24804]

Kris, any thoughts on what would cause us to have 24,804 structured clone holders around?
Flags: needinfo?(kmaglione+bmo)
(In reply to Eric Rahm [:erahm] (please no mozreview requests) from comment #39)
> This is pretty odd:
> 
> > ├────274.58 MB (24.19%) -- dom
> > │    ├──274.17 MB (24.16%) ── structured-clone-holder [24804]
> 
> Kris, any thoughts on what would cause us to have 24,804 structured clone
> holders around?

This has shown up in other bugs too, but I haven't been able to reproduce it in newer Firefox versions. It would be nice to know what Firefox versions these memory reports are from.

In the past, the problem was that extensions would send tons of messages, promise to send a response to them, and then not send a response. That led to the StructuredCloneHolders that held the message data staying alive until the page the messages were sent to closed. Ghostery tends to be particularly bad about this.

But we fixed that problem a few releases ago by adding finalization witnesses to the response callbacks and nulling out the StructuredCloneHolder references as soon as we deserialized them.
Flags: needinfo?(kmaglione+bmo)
I'm assuming this is 59 but we might as well ask. Jerry, what version are you running? It looks like the issue might be related to Ghostery, so I'd suggest disabling that for now.
Flags: needinfo?(jbullis)
I have the problem with zerohedge.com. No Ghostery or any other user installed plugins. Only plugins "OpenH264 Video Codec provided by Cisco Systems, Inc.", "Widevine Content Decryption Module provided by Google Inc.", "Shockwave Flash". Looks this is the basic set.

In comment 33, I read "I filed bug 1428239 for zerohedge.com."

But the comment refers to this bug 1428239 not the other bug.

The other bug is bug 1451517 and not referring to a memory leak. I filed bug 1455161 for the memory leak.
That memory map is from Firefox version 59.0.2 64 bit.
Flags: needinfo?(jbullis)
"OpenH264 Video Codec provided by Cisco Systems, Inc." is also one of my add-ons.  But as far as I know it doesn't do any outside communication.  I would also say that currently the Windows 7 machine is experiencing a slow memory leak.  My Windows 10 laptop is having a major leak from time to tine.  At this time, I haven't been able to catch it soon enough to capture a memory map.  Interestingly enough, this morning, things weren't looking well when I woke it up.  Had task manager up on the screen when it went to sleep and it looked normal except that all Firefox processes showed as having ZERO memory.  The system was so tied up, I just killed Firefox and things cleared up in a couple of minutes without rebooting.  Lots of paging during the recovery, so I betting Firefox had the memory tied up again.
Priority: -- → P3
About:Memory file from laptop where Firefox memory use was slowing PC
Attached file my memory dump
Firefox Developer (61.0b6)
Over the last few patches the memory problem has gotten better.  The last two patches, I haven't noticed a problem with memory leaks.  At this point I think you can close this problem.
Thanks for the update. I'm sorry we weren't able to more directly help.
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: