Andrei Petcu Reporter
	Description • 9 years ago

User Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:38.0) Gecko/20100101 Firefox/38.0
Build ID: 20150605133456

Steps to reproduce:

1) install this js app from web
git clone git@github.com:andreicristianpetcu/foobar.git
cd foobar
git checkout ef75f84
npm install -g grunt
npm install
bower install
grunt server

go to http://localhost:8000 and either run in the cli "touch app/index.html" or click the first link in the page several times.


Actual results:

The DevTools network tab shows the connection is really slow, it loads the whole page in 4 seconds while Chromium loads the page in 532 milliseconds. Firefox Nightly loads it a little faster than the stable version in around 3.5 seconds (probably due to bug 1143224 being fixed). I think this issue is more related to the large number of connections, but this is a normal JavaScript project these days, it is not a huge project.

Developers rely on liveload a lot these days and a slow reloading of the page really breaks the workflow.

Here is a video (watch in fullhd) https://youtu.be/zbp2syKiyj0
I made a demo page http://andreicristianpetcu.github.io/foobar/app/ but it does reproduce the bug as well as the installed version :(


Expected results:

The page should load in under a second. Please do not mark this bug as duplicate of 1143224 :P since it is not. I tested with a Nightly that already has the fix for that bug.

	Andrei Petcu Reporter
	Comment 1 • 9 years ago

If anybody is interested I plan to put a small bounty on this bug. I know I cannot cover the cost of this fix but I hope more and more people start putting bounties on features and bugs and make Firefox more independent.

	Brian Grinstead [:bgrins]
	Comment 2 • 9 years ago

I can confirm that it's slower in FF than chrome.  I'm seeing something like ~2 seconds in FF (on an fx-team build with a clean profile) and ~1 second on Chrome canary in a clean profile.

	Andrei Petcu Reporter
	Comment 3 • 9 years ago

Awesome! Hope it's fixable :D

	Jeff Griffiths (:canuckistani) (:⚡︎)
	Comment 4 • 9 years ago

(In reply to andrei from comment #0)
> User Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:38.0) Gecko/20100101
> Firefox/38.0
> Build ID: 20150605133456
> 
> Steps to reproduce:
> ...

This is an amazing bug report - thanks so much for taking the time!

	Andrei Petcu Reporter
	Comment 5 • 9 years ago

I've put a modest 15$ bounty on this bug :) https://www.bountysource.com/issues/22407575-firefox-liveload-with-lots-of-connections-slow-when-netmonitor-is-open

	Andrei Petcu Reporter
	Comment 6 • 9 years ago

Do you have an estimation when this bug will be fixed or if it can be fixed?

If this issue gets fixed and lands in Nightly or Aurora before #1143224 lands in stable I can make a video with the performance gain of both bugs :D

I think lots of JS developers switched to other browsers because of this. I know how annoyed my work colleagues are when they work on my computer and brag on how fast "their" browser is :P

Thank you!

	Andrei Petcu Reporter
	Comment 7 • 9 years ago

I've just put another 30$ on this bounty. It is a total of 45$ now.

I guess this is peanuts for a software developer (especially if he or she is living in the US) but if more people donated small amounts as I do, maybe Firefox can become more independent.

I really want this bug fixed, and I do not have the skills to fix it myself. This really breaks my flow since I loose a few seconds each time I save a file in my web project.

I hope some Mozilla developers will have time for this bug soon :)

	Andrei Petcu Reporter
	Comment 8 • 9 years ago

I ran this with a fresh profile, no addons, no AdBlock.

	Dan Callahan [:callahad]
	Comment 9 • 9 years ago

Adding DevAdvocacy keyword since there's some social proof of this being a real issue for folks on Reddit: https://www.reddit.com/r/firefox/comments/3ghmba/firefox_is_really_slow_with_liveload_i_loose_a/

Andrei: Thank you so much for the detailed report and test case. I'll take that over a bounty any day. :)

	petre.2dor
	Comment 10 • 9 years ago

Hi, 
I was also able to replicate this. Hope it gets some attention :)

Cheers,
Petre

	Jeff Griffiths (:canuckistani) (:⚡︎)
	Comment 11 • 9 years ago

Brian you were the last person in the netmonitor IIRC - can you estimate difficulty for this?

	Brian Grinstead [:bgrins]
	Comment 12 • 9 years ago

(In reply to Jeff Griffiths (:canuckistani) from comment #11)
> Brian you were the last person in the netmonitor IIRC - can you estimate
> difficulty for this?

Running in e10s does seem to help a bit (maybe half a second faster), which makes me think there's still some frontend jank but it isn't fixing the bulk of the problem.  And there probably isn't a ton more low hanging fruit for that after Bug 1143224.

Getting this to have (near)zero impact would be hard, we are always going to have some overhead for platform level monitoring and RDP traffic.  But I think if we put some effort towards profiling this we should be able to get better than today with less work than that.

I'm seeing something weird in the waterfall on the demo where there is a long delay before the recaptcha__en.js request happens.  It looks like nearly all of the requests are done in < .5 seconds, but that one doesn't start until 2.5 seconds in.  I see a similar thing on a full reload, but it's not as pronounced.  I'll upload a HAR file with more info.

	Brian Grinstead [:bgrins]
	Comment 13 • 9 years ago

HAR export - this can viewed by dropping it onto http://www.softwareishard.com/har/viewer/

	J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)
	Comment 14 • 9 years ago

Not sure if this was already the case during your last profile, but I noticed that desktop APZ (which defaults on in Nightly for Mac) makes scrolling the netmonitor much jankier and slower.  Disabling APZ makes this better.

Not sure if there's anything the tool itself can do for this piece of it.

	J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)
	Comment 15 • 9 years ago

(In reply to J. Ryan Stinnett [:jryans] (use ni?) from comment #14)
> Not sure if this was already the case during your last profile, but I
> noticed that desktop APZ (which defaults on in Nightly for Mac) makes
> scrolling the netmonitor much jankier and slower.  Disabling APZ makes this
> better.

I filed bug 1201192 about the APZ issues with net monitor.

	Andrei Petcu Reporter
	Comment 16 • 9 years ago

Is there any update on this bug? I see it's a P1, does this mean it has high priority or low priority? Thank you!

	Jan Honza Odvarko [:Honza] (always need-info? me)
	Comment 17 • 8 years ago

I have been testing this and I can also reproduce the problem.

I tried two scenarios:

1) Loading the page *with* the toolbox open (having the Net panel selected or unselected didn't make visible impact). The loading time in this scenario was ~3-4 sec, which corresponds to comment #0.

2) Loading the page *without* the toolbox open, but having HARTriggerExport extension [1] installed so, I can collect HAR timing info. The loading time in this scenarion was ~1.3 sec, which is equal to what I am seeing in Chromium.

[1] http://www.softwareishard.com/blog/har-export-trigger/

These results clearly points to the Net panel UI. It looks like the Network panel logic or rather the Network panel rendering is too expensive and affects the overall page load timing - slowing it significantly down.

I'd really like to work on this, but I have too much other priorities on my plate at the moment and so, I am lowering the priority to #2 - just because I don't know about anyone who could work on this now. 

Also adopting React in the Net panel, which is already planned, could help to solve this problem. React is fast and better equipped for rendering optimizations (comparing to XUL).

---

Just a note for the scenario #2, you need to:
devtools.netmonitor.har.enableAutoExportToFile = true
extensions.netmonitor.har.autoConnect = true
+ restart the browser

Honza

	Andrei Petcu Reporter
	Comment 18 • 8 years ago

When is the switch to React planned? Can it be done incrementally (only some dev tools parts at first and then the rest)?

	Jan Honza Odvarko [:Honza] (always need-info? me)
	Comment 19 • 8 years ago

We are already switching to react (see bug 1263741 as one of the meta bugs), focusing on the Inspector panel atm. The Net panel will follow.

Honza

	Andrei Petcu Reporter
	Comment 20 • 8 years ago

Can you point me to the specific bugs that target the migration of the network panel to React? I want to put a bounty on their heads :D I did not manage to find them myself :P

	Jan Honza Odvarko [:Honza] (always need-info? me)
	Comment 21 • 8 years ago

I don't think there are reports specifically for the Net panel.
So far, we have meta bug 1263750

Honza

	Jarda Snajdr [:jsnajdr]
	Comment 22 • 8 years ago

When the Netmonitor frontend receives an event like "responseContent", it immediately fetches the response data with webConsoleClient.getResponseContent(), although this data is not needed to display the request list. It's only needed to show the request detail. So, there is a lot of useless RDP traffic.

Making this fetching lazy immediately cuts the load time to half, from 4s to 2s. Chrome is still far better at 0.5s, so I hope there are more low-hanging optimizations like this.

	Andrei Petcu Reporter
	Comment 23 • 8 years ago

This is really nice! Is this lazy loading in Nightly? Can I test it out? :D

	Jarda Snajdr [:jsnajdr]
	Comment 24 • 8 years ago

(In reply to andrei from comment #23)
> This is really nice! Is this lazy loading in Nightly? Can I test it out? :D

Right now, it's just an unfinished patch on my machine. You'll be able to test a try build when I start testing this, and it'll get to Nightly only when it lands and gets RESOLVED FIXED.

	Andrei Petcu Reporter
	Comment 25 • 8 years ago

cool. I can test the try build when you make it.

	Brian Grinstead [:bgrins]
	Comment 26 • 8 years ago

(In reply to Jarda Snajdr [:jsnajdr] from comment #22)
> Making this fetching lazy immediately cuts the load time to half, from 4s to
> 2s. Chrome is still far better at 0.5s, so I hope there are more low-hanging
> optimizations like this.

Great find, looking forward to this improvement!

	Jarda Snajdr [:jsnajdr]
	Comment 27 • 8 years ago

First working version of the lazy-loading patch. In my testing, it cuts the loading time to half compared to the non-patched version.

Almost all mochitests are passing, except HAR and filtering ones.

It will require a lot of cleanup and dividing into multiple reviewable patches, but is ready to be tested.

	Jarda Snajdr [:jsnajdr]
	Comment 28 • 8 years ago

https://treeherder.mozilla.org/#/jobs?repo=try&revision=e063269f32e8

	Jarda Snajdr [:jsnajdr]
	Comment 29 • 8 years ago

Andrei, when the try build in comment 28 finishes, you can download the linux64 binaries and try out how the fix works for you.

	Andrei Petcu Reporter
	Comment 30 • 8 years ago

So I need to download from here https://archive.mozilla.org/pub/firefox/try-builds/jsnajdr@gmail.com-34b97e789e5b1e9999c87a826f745caff080e0c9/try-linux64/ this https://archive.mozilla.org/pub/firefox/try-builds/jsnajdr@gmail.com-34b97e789e5b1e9999c87a826f745caff080e0c9/try-linux64/firefox-51.0a1.en-US.linux-x86_64.tar.bz2 right?

	Brian Grinstead [:bgrins]
	Comment 31 • 8 years ago

(In reply to andrei from comment #30)
> So I need to download from here
> https://archive.mozilla.org/pub/firefox/try-builds/jsnajdr@gmail.com-
> 34b97e789e5b1e9999c87a826f745caff080e0c9/try-linux64/ this
> https://archive.mozilla.org/pub/firefox/try-builds/jsnajdr@gmail.com-
> 34b97e789e5b1e9999c87a826f745caff080e0c9/try-linux64/firefox-51.0a1.en-US.
> linux-x86_64.tar.bz2 right?

Yes

	Andrei Petcu Reporter
	Comment 32 • 8 years ago

Are you sure? I get the same performance on both Nightly and the try build. Each one has a brand new profile so there are no addons messing around with my data.

The page in the project above (foobar) is ~1.7s for both builds as an average and I get 1.5 best and 2s worst but the same data is for both of them. I tried both manually loading each page with F5 so that the two builds don't mess with each others data and network and I also used the "touch app/index.html" command.

Am I doing something wrong? The try build did not update itself, I did not restart it.

	Andrei Petcu Reporter
	Comment 33 • 8 years ago

There has been some other things improving the performance besides your work. My initial bug report showed 3.7s on that nightly but today's nightly is 1.7s.

	Andrei Petcu Reporter
	Comment 34 • 8 years ago

I run my FF instances with:

~/Downloads/firefox_liveload/firefox -P "vanilla_fast_connections" -new-window

~/firefox_nightly/firefox -P "vanilla" -new-window

	Jarda Snajdr [:jsnajdr]
	Comment 35 • 8 years ago

(In reply to andrei from comment #32)
> Are you sure? I get the same performance on both Nightly and the try build.
> Each one has a brand new profile so there are no addons messing around with
> my data.

Hmm, do you have e10s enabled? I now realize that the patch only improves performance on e10s. It avoids unnecessary transfers of large data, and that has impact only if the transfer has significant cost. I.e., moving the data across process boundaries. On non-e10s, the transfer costs almost nothing.

By the way, my latest unpatched Nightly is also significantly faster than few days ago. I'm wondering what caused that.

	Andrei Petcu Reporter
	Comment 36 • 8 years ago

e10s is default in Nightly so yes.... I use it.

	Jarda Snajdr [:jsnajdr]
	Comment 37 • 8 years ago

(In reply to andrei from comment #36)
> e10s is default in Nightly so yes.... I use it.

The try build has a lot of test failures that shouldn't be there - these tests succeeded locally. So there's something wrong with the pushed patch. I'll need to fix that and then try again.

	Brian Grinstead [:bgrins]
	Comment 38 • 8 years ago

(In reply to andrei from comment #34)
> I run my FF instances with:
> 
> ~/Downloads/firefox_liveload/firefox -P "vanilla_fast_connections"
> -new-window
> 
> ~/firefox_nightly/firefox -P "vanilla" -new-window

You might need to do -no-remote (or close out any other running process) to make sure it's actually loading the try build and not using an existing process.

	Jan Honza Odvarko [:Honza] (always need-info? me)
	Comment 39 • 8 years ago

I have been also testing this using STR from comment #0 (on my own build, Win10)

Unfortunately, I couldn't see any differences in page-load-speed when testing with/without the patch applied. Simple re-load of the page (pressing F5 or modifying foobar/app/index.html) takes about 2.5 s on my machine. I tried to switch on/off e10s, but didn't observe any changes...

Honza

	Jarda Snajdr [:jsnajdr]
	Comment 40 • 8 years ago

(In reply to Jan Honza Odvarko [:Honza] from comment #39)
> Unfortunately, I couldn't see any differences in page-load-speed when
> testing with/without the patch applied. Simple re-load of the page (pressing
> F5 or modifying foobar/app/index.html) takes about 2.5 s on my machine. I
> tried to switch on/off e10s, but didn't observe any changes...

Very strange, I stopped seeing any improvements, too. :(

	Andrei Petcu Reporter
	Comment 41 • 7 years ago

Firefox gets a bad rep due to this DevTools bug when people do benchmarks :(

I seen it several times, not just in this video.

https://youtu.be/ULnd_PoW-kY?t=140