TRR: Infinite loop when visiting https://1.1.1.1/help
Categories
(Core :: Networking: DNS, defect, P1)
Tracking
()
People
(Reporter: marek, Assigned: valentin)
Details
(Whiteboard: [necko-triaged])
Attachments
(4 files)
9.60 KB,
text/x-log
|
Details | |
47 bytes,
text/x-phabricator-request
|
jcristau
:
approval-mozilla-beta+
|
Details | Review |
47 bytes,
text/x-phabricator-request
|
jcristau
:
approval-mozilla-beta+
|
Details | Review |
47 bytes,
text/x-phabricator-request
|
jcristau
:
approval-mozilla-beta+
|
Details | Review |
User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:72.0) Gecko/20100101 Firefox/72.0
Steps to reproduce:
Enable TRR with default configuration and go to https://1.1.1.1/help
Actual results:
The site loads but FF starts consuming 100% CPU and remains to do so even after closing the tab. It seems to be asking for ipv6{a,b}.cloudflare-dns.com in a loop.
Expected results:
If it's a problem with javascript, I'd expect it to go away after closing a tab, but nothing shows up in the performance console.
Assignee | ||
Comment 1•4 years ago
|
||
Doesn't seem to happen if the network doesn't have IPv6 connectivity. I'll check it again next week when I find a proper network.
Thanks! This seems to happen both on OS X and Linux, on 72 and nightly. It does seem to happen only on networks with IPv6 connectivity.
Assignee | ||
Comment 3•4 years ago
|
||
So, indeed, this bug only reproduces when the page displays: 2606:4700:4700::1111 Yes
in the Connectivity to Resolver IP Addresses
section - so only when the network has some kind of IPv6 connectivity.
There are two things I've noticed happening here:
- This profile https://perfht.ml/2RTjZ5N shows a kind of infinite loop in WebRequest.jsm::getInterface
- I also caught this in rr, and when looking at the stacktrace of the DNS resolution for the
ipv6{a,b}.cloudflare-dns.com
domains, I see the channels are triggered fromPerformBackgroundCacheRevalidationNow
. Commenting out this method seems to fix the issue.
Honza, can I get your thoughts on no.2 ? I'll keep trying to figure out why no.1 happens. I'm quite sure that's also a factor in this bug.
Comment 4•4 years ago
|
||
(In reply to Valentin Gosu [:valentin] (he/him) from comment #3)
- I also caught this in rr, and when looking at the stacktrace of the DNS resolution for the
ipv6{a,b}.cloudflare-dns.com
domains, I see the channels are triggered fromPerformBackgroundCacheRevalidationNow
. Commenting out this method seems to fix the issue.Honza, can I get your thoughts on no.2 ? I'll keep trying to figure out why no.1 happens. I'm quite sure that's also a factor in this bug.
This can happen only when the response has Cache-control: stale-while-revalidate
response header.
I will have to try to reproduce this.
Assignee | ||
Comment 5•4 years ago
|
||
I've managed to reproduce this without TRR turned on.
STR:
Have the secure-proxy addon turned on.
Go to https://1.1.1.1/help
The connectivity to the IPv6 addresses should say Yes.
Turn off the secure proxy and refresh the page.
You'll see a ton of revalidation requests happening in the background.
(Note, I had to replace the MOZ_ASSERT here with a NS_ASSERTION to avoid crashes in debug more)
Assignee | ||
Comment 6•4 years ago
|
||
These are the headers for the https://ipv6a.cloudflare-dns.com/resolvertest
requests
2020-02-26 13:38:35.164389 UTC - [Parent 7541: Cache2 I/O]: D/nsHttp nsHttpResponseHead::ParseVersion [version=HTTP/2 200 OK
date: Wed, 26 Feb 2020 13:34:03 GMT
content-type: text/plain; charset=utf-8
content-length: 1
access-control-allow-origin: *
access-control-allow-methods: GET, OPTIONS
access-control-expose-headers: Content-Length,Content-Range,CF-RAY
timing-allow-origin: *
cache-control: s-maxage=86400, stale-while-revalidate=86400, immutable
cf-cache-status: DYNAMIC
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
alt-svc: h3-25=":443"; ma=86400, h3-24=":443"; ma=86400, h3-23=":443"; ma=86400
strict-transport-security: max-age=31536000; includeSubDomains; preload
x-content-type-options: nosniff
server: cloudflare
cf-ray: 56b24b530b4ddabc-ARN
X-Firefox-Spdy: h2
X-Firefox-Spdy-Proxy: true
]
Assignee | ||
Comment 7•4 years ago
|
||
Updated•4 years ago
|
Assignee | ||
Updated•4 years ago
|
Assignee | ||
Updated•4 years ago
|
Assignee | ||
Comment 8•4 years ago
|
||
Depends on D64397
Assignee | ||
Comment 9•4 years ago
|
||
Depends on D65813
Updated•4 years ago
|
Comment 10•4 years ago
|
||
Pushed by valentin.gosu@gmail.com: https://hg.mozilla.org/integration/autoland/rev/75575bddf6a4 Add test for stale-while-revalidate loop bug https://hg.mozilla.org/integration/autoland/rev/74c573da7fa7 Add pref for stale-while-revalidate r=dragana https://hg.mozilla.org/integration/autoland/rev/42e1f8160f4c Make sure channels doing a stale-while-revalidate cannot trigger another revalidation r=dragana
Comment 12•4 years ago
|
||
bugherder |
https://hg.mozilla.org/mozilla-central/rev/75575bddf6a4
https://hg.mozilla.org/mozilla-central/rev/74c573da7fa7
https://hg.mozilla.org/mozilla-central/rev/42e1f8160f4c
Assignee | ||
Comment 13•4 years ago
|
||
Comment on attachment 9131591 [details]
Bug 1611651 - Make sure channels doing a stale-while-revalidate cannot trigger another revalidation r=dragana
Beta/Release Uplift Approval Request
- User impact if declined: When visiting pages that use the "stale-while-revalidate" header we get into a loop and use 100% of CPU
- Is this code covered by automated tests?: Yes
- Has the fix been verified in Nightly?: Yes
- Needs manual test from QE?: Yes
- If yes, steps to reproduce: see comment 5
- List of other uplifts needed: None
- Risk to taking this patch: Low
- Why is the change risky/not risky? (and alternatives if risky): The fix is trivial. It just adds a bool to break the loop.
We also add a pref to disable the feature completely if needed. - String changes made/needed:
Assignee | ||
Updated•4 years ago
|
Comment 14•4 years ago
|
||
Comment on attachment 9129200 [details]
Bug 1611651 - Add test for stale-while-revalidate loop bug r=dragana
necko fix for 75.0b10
Updated•4 years ago
|
Updated•4 years ago
|
Comment 15•4 years ago
|
||
bugherder uplift |
https://hg.mozilla.org/releases/mozilla-beta/rev/33cb6654c64b
https://hg.mozilla.org/releases/mozilla-beta/rev/f39beaef9ee1
https://hg.mozilla.org/releases/mozilla-beta/rev/a22e626aa75f
Description
•