Closed Bug 1611651 Opened 4 years ago Closed 4 years ago

TRR: Infinite loop when visiting https://1.1.1.1/help

Categories

(Core :: Networking: DNS, defect, P1)

72 Branch
defect

Tracking

()

RESOLVED FIXED
mozilla76
Tracking Status
firefox75 --- fixed
firefox76 --- fixed

People

(Reporter: marek, Assigned: valentin)

Details

(Whiteboard: [necko-triaged])

Attachments

(4 files)

Attached file trr.debug.log

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:72.0) Gecko/20100101 Firefox/72.0

Steps to reproduce:

Enable TRR with default configuration and go to https://1.1.1.1/help

Actual results:

The site loads but FF starts consuming 100% CPU and remains to do so even after closing the tab. It seems to be asking for ipv6{a,b}.cloudflare-dns.com in a loop.

Expected results:

If it's a problem with javascript, I'd expect it to go away after closing a tab, but nothing shows up in the performance console.

Doesn't seem to happen if the network doesn't have IPv6 connectivity. I'll check it again next week when I find a proper network.

Flags: needinfo?(valentin.gosu)

Thanks! This seems to happen both on OS X and Linux, on 72 and nightly. It does seem to happen only on networks with IPv6 connectivity.

So, indeed, this bug only reproduces when the page displays: 2606:4700:4700::1111 Yes in the Connectivity to Resolver IP Addresses section - so only when the network has some kind of IPv6 connectivity.

There are two things I've noticed happening here:

  1. This profile https://perfht.ml/2RTjZ5N shows a kind of infinite loop in WebRequest.jsm::getInterface
  2. I also caught this in rr, and when looking at the stacktrace of the DNS resolution for the ipv6{a,b}.cloudflare-dns.com domains, I see the channels are triggered from PerformBackgroundCacheRevalidationNow. Commenting out this method seems to fix the issue.

Honza, can I get your thoughts on no.2 ? I'll keep trying to figure out why no.1 happens. I'm quite sure that's also a factor in this bug.

Flags: needinfo?(valentin.gosu) → needinfo?(honzab.moz)

(In reply to Valentin Gosu [:valentin] (he/him) from comment #3)

  1. I also caught this in rr, and when looking at the stacktrace of the DNS resolution for the ipv6{a,b}.cloudflare-dns.com domains, I see the channels are triggered from PerformBackgroundCacheRevalidationNow. Commenting out this method seems to fix the issue.

Honza, can I get your thoughts on no.2 ? I'll keep trying to figure out why no.1 happens. I'm quite sure that's also a factor in this bug.

This can happen only when the response has Cache-control: stale-while-revalidate response header.

I will have to try to reproduce this.

I've managed to reproduce this without TRR turned on.

STR:
Have the secure-proxy addon turned on.
Go to https://1.1.1.1/help
The connectivity to the IPv6 addresses should say Yes.
Turn off the secure proxy and refresh the page.
You'll see a ton of revalidation requests happening in the background.

(Note, I had to replace the MOZ_ASSERT here with a NS_ASSERTION to avoid crashes in debug more)

These are the headers for the https://ipv6a.cloudflare-dns.com/resolvertest requests

2020-02-26 13:38:35.164389 UTC - [Parent 7541: Cache2 I/O]: D/nsHttp nsHttpResponseHead::ParseVersion [version=HTTP/2 200 OK
date: Wed, 26 Feb 2020 13:34:03 GMT
content-type: text/plain; charset=utf-8
content-length: 1
access-control-allow-origin: *
access-control-allow-methods: GET, OPTIONS
access-control-expose-headers: Content-Length,Content-Range,CF-RAY
timing-allow-origin: *
cache-control: s-maxage=86400, stale-while-revalidate=86400, immutable
cf-cache-status: DYNAMIC
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
alt-svc: h3-25=":443"; ma=86400, h3-24=":443"; ma=86400, h3-23=":443"; ma=86400
strict-transport-security: max-age=31536000; includeSubDomains; preload
x-content-type-options: nosniff
server: cloudflare
cf-ray: 56b24b530b4ddabc-ARN
X-Firefox-Spdy: h2
X-Firefox-Spdy-Proxy: true
]
Assignee: nobody → valentin.gosu
Assignee: valentin.gosu → honzab.moz
Status: UNCONFIRMED → NEW
Ever confirmed: true
Priority: -- → P1
Whiteboard: [necko-triaged]
Assignee: honzab.moz → valentin.gosu
Attachment #9129200 - Attachment description: Bug 1611651 - Add test for stale-while-revalidate loop bug r=mayhemer → Bug 1611651 - Add test for stale-while-revalidate loop bug r=dragana
Pushed by valentin.gosu@gmail.com:
https://hg.mozilla.org/integration/autoland/rev/75575bddf6a4
Add test for stale-while-revalidate loop bug
https://hg.mozilla.org/integration/autoland/rev/74c573da7fa7
Add pref for stale-while-revalidate r=dragana
https://hg.mozilla.org/integration/autoland/rev/42e1f8160f4c
Make sure channels doing a stale-while-revalidate cannot trigger another revalidation r=dragana

Thanks for taking care of this bug, Valentin!

Flags: needinfo?(honzab.moz)
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla76

Comment on attachment 9131591 [details]
Bug 1611651 - Make sure channels doing a stale-while-revalidate cannot trigger another revalidation r=dragana

Beta/Release Uplift Approval Request

  • User impact if declined: When visiting pages that use the "stale-while-revalidate" header we get into a loop and use 100% of CPU
  • Is this code covered by automated tests?: Yes
  • Has the fix been verified in Nightly?: Yes
  • Needs manual test from QE?: Yes
  • If yes, steps to reproduce: see comment 5
  • List of other uplifts needed: None
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): The fix is trivial. It just adds a bool to break the loop.
    We also add a pref to disable the feature completely if needed.
  • String changes made/needed:
Attachment #9131591 - Flags: approval-mozilla-beta?
Attachment #9129200 - Flags: approval-mozilla-beta?
Attachment #9131590 - Flags: approval-mozilla-beta?

Comment on attachment 9129200 [details]
Bug 1611651 - Add test for stale-while-revalidate loop bug r=dragana

necko fix for 75.0b10

Attachment #9129200 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
Attachment #9131590 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
Attachment #9131591 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: