Closed Bug 1558167 Opened 5 years ago Closed 5 years ago

[10.15] WebRender broken on macOS 10.15 Catalina

Categories

(Core :: Graphics: WebRender, defect, P3)

All
macOS
defect

Tracking

()

RESOLVED FIXED
mozilla71
Tracking Status
firefox-esr60 --- unaffected
firefox-esr68 --- disabled
firefox67 --- disabled
firefox68 --- disabled
firefox69 --- disabled
firefox70 --- disabled
firefox71 --- fixed

People

(Reporter: Panajev, Assigned: kvark)

References

(Blocks 1 open bug)

Details

(Keywords: regression)

Attachments

(17 files)

840.54 KB, image/png
Details
86.23 KB, image/png
Details
14.69 KB, text/plain
Details
14.91 KB, text/plain
Details
573.37 KB, image/png
Details
981.23 KB, image/png
Details
1.10 MB, image/png
Details
14.34 KB, text/plain
Details
131.92 KB, image/png
Details
157.46 KB, image/png
Details
47 bytes, text/x-phabricator-request
Details | Review
16.95 KB, text/plain
Details
12.39 KB, text/plain
Details
1.12 MB, image/png
Details
1.14 MB, image/png
Details
47 bytes, text/x-phabricator-request
Details | Review
8.11 KB, text/plain
Details

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:69.0) Gecko/20100101 Firefox/69.0
Firefox for Android

Steps to reproduce:

Install macOS 10.15 Catalina on MacBook Pro mid 2018 (which was running Firefox Nightly with WebRender enabled through about:config)

Install Firefox Nightly and enabled Web Render

Browse the web

Actual results:

Text corruption, images rendering corruption

Expected results:

I expected the same HW to be able to run the same version of Firefox Nightly with the same WebRender renderer active and drawing to screen correctly

Doesn't look like this needs to be sec-sensitive.

Group: firefox-core-security
Component: Untriaged → Graphics
Product: Firefox → Core
Blocks: wr-mac
Component: Graphics → Graphics: WebRender
Priority: -- → P3
Summary: WebRender broken on macOS 10.15 Catalina → [10.15] WebRender broken on macOS 10.15 Catalina

Hello Goffredo - I have WR enabled on my MacBook Pro running 10.15 and I don't see any corruption. We probably need more information about your hardware to try to debug this issue. Also can you confirm which about:config pref you toggled? Thanks!

Flags: needinfo?(Panajev)

When everything is corrupted, please open about:support, click on "Copy text to clipboard" (second button top left), paste it into a text file and upload it here (Attach file). Thanks!

(In reply to Marcia Knous [:marcia - needinfo? me] from comment #2)

Hello Goffredo - I have WR enabled on my MacBook Pro running 10.15 and I don't see any corruption. We probably need more information about your hardware to try to debug this issue. Also can you confirm which about:config pref you toggled? Thanks!

Hello, the preference is: gfx.webrender.all

HW is: attached screenshot for HW

Flags: needinfo?(Panajev)
Attached image HW info

Got an entire tab, page rendering included not just the browser UI, corrupted

I can confirm the same issue. Firefox 68.0b13 (64-bit). MacOS 10.15 Beta (19A487m).

I'm seeing the same — glyphs rendered as red/green/cyan blobs, and it gets progressively worse (seconds after restart some text is ok, by the time I got to about:config it was all unreadable).

Setting gfx.webrender.force-disabled=true makes everything look normal again.

^ Firefox nightly 69.0a1 (2019-07-08) (64-bit) on Catalina 10.15 Beta (19A501i) on MBP with Radeon Pro 560X 4 GB + Intel UHD Graphics 630 1536 MB

Additional data point:

MBP 2016 (Radeon Pro 450 2 GB), connected to an external 4K monitor. This is with the settings gfx.webrender.all enabled. All my add-ons were disabled for this test. Next attachment shows the webrender setting disabled.

From comment above. This is when restarting the browser in safe mode, which I assume ignores the webrender config and defaults to false (not sure about that though).

Attached image screenshot

This is how it looks for me with WebRender enabled on macOS Catalina Beta on a MacBook Pro from late 2018 with Radeon Pro 560X.

I also have this problem on macOS 10.15 Public Beta 9.

Resetting these two settings to their default of "false" fixes it:

gfx.webrender.all = false
gfx.webrender.enabled = false

This was really concerning until I remembered I had set the above to true

The problem showed up on the second or third restart after upgrading the OS -- the initial launch worked okay.

Attached about:support and two screenshot examples.

attachment 9097082 [details]
attachment 9097083 [details]
attachment 9097084 [details]

Status: UNCONFIRMED → NEW
Ever confirmed: true
OS: Unspecified → macOS
Hardware: Unspecified → All
Version: 69 Branch → Trunk

Dzmitry, can you file a bug with Apple about this issue?

Flags: needinfo?(dmalyshau)

Could be AMD-specific? I'm not able to reproduce this on macOS 10.15 (19A582a) with MBP 2016, Intel Iris 550 graphics.

Waiting for Markus to update the AMD-based MBP and reproduce locally...

Flags: needinfo?(dmalyshau)

Is there something we need to do here for 70 before release? Tracking for now, in case.

(In reply to Dzmitry Malyshau [:kvark] from comment #20)

Waiting for Markus to update the AMD-based MBP and reproduce locally...

It turns out that machine does not have an AMD GPU. We still haven't identified a machine on which to try reproducing this in the Toronto office.

Adding Camelia to see if they can reproduce on the QA side.

Flags: needinfo?(camelia.badau)

Changing the priority to p1 as the bug is tracked by a release manager for the current beta.
See What Do You Triage for more information

Priority: P3 → P1

(In reply to Liz Henry (:lizzard) from comment #22)

Is there something we need to do here for 70 before release? Tracking for now, in case.

No. WebRender is not enabled on Mac in 70. To run into this a user would need to gfx.webrender.all

Priority: P1 → P3

We have a machine now.

Flags: needinfo?(camelia.badau)
Assignee: nobody → dmalyshau
Status: NEW → ASSIGNED
Pushed by dmalyshau@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/2e23a503638e
Disable swizzling on macOS 10.15 AMD devices r=jrmuizel
Status: ASSIGNED → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla71

Assuming that the 20191009095806 nightly includes 2e23a503638e, this doesn't fix the issues for me.

Dirkjan, could you provide your about:support?

Status: RESOLVED → REOPENED
Flags: needinfo?(dirkjan)
Resolution: FIXED → ---
Attached file about:support for :djc
Flags: needinfo?(dirkjan)
Attached file about:support
@Dzmitry: Under 20191009103354, it seems to be affecting the browser chrome only, web pages are working fine with webrender.all set to true.

Wow, that's weird! Could you try running the artifact from a try push that completely disables the swizzling? Don't forget to ensure WR is ON, please.

Also, please provide a screenshot of how it works under 20191009103354 for you.

Flags: needinfo?(cumanzor)

This on build 20191009103354. I'll test the artifact in a sec.

Flags: needinfo?(cumanzor)

@Dzmitry: seems to be glitching out a bit under build 20191008171937 as well. The tab bar changes color from blue to white when switching tabs, and the tab title is a bit glitched out.

I can repro. Looking into it...
It's possible that Swizzling is not the only thing that's going wrong here.

Here is something interesting. Playing with the blob example I can see that:

  1. it is reproducible, i.e. I see visual issues with the colors, and I can record the GL stream with apitrace and then replay it on Intel with no issues
  2. the issue shows up in the texture upload stage - this is before we do anything with the actual swizzle configuration of texture units (!)
  3. out of 4 uploaded chunks, only one that is not a power of two size gets buggy results

This leads me to the following hypothesis: the AMD bug is not caused by swizzling, it's in the texture uploads, related to how pixel unpack buffer is read. When swizzling is forcefully disabled, the driver has to convert the format (from BGRA to RGBA), and thus it's going a completely different code path that is not affected by the bug.

For an extra test, I disabled all the swizzle setting code (on texture units), and I still got an issue. That somewhat aligns with the observation that there are still visual issues even if swizzling is disabled in Gecko - because it's not the swizzling fault.

Bugbug thinks this bug is a regression, but please revert this change in case of error.

Keywords: regression
Pushed by dmalyshau@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/8d93170e3aaf
Force 256 alignment for texture uploads on AMD on macOS r=jrmuizel

Comment on attachment 9100208 [details]
Force 256 alignment for texture uploads on AMD on macOS

Beta/Release Uplift Approval Request

  • User impact if declined: Some of the macOS users who force-enable WR will find it unusable.
  • Is this code covered by automated tests?: Yes
  • Has the fix been verified in Nightly?: No
  • Needs manual test from QE?: No
  • If yes, steps to reproduce:
  • List of other uplifts needed: None
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): Risk is low, because it is specifically written to only affect a small population of WR on macOS.
  • String changes made/needed:
Attachment #9100208 - Flags: approval-mozilla-beta?
Attachment #9099661 - Flags: approval-mozilla-beta?
Status: REOPENED → RESOLVED
Closed: 5 years ago5 years ago
Resolution: --- → FIXED

How can we track when this reaches the Beta channel?

The request for approval has been filed in Comment 43, you'll see the follow-up comments in here.

Dzmitry, is there some way we can verify this fix and can you describe it for QA?
This is very last minute for uplift to 70, since the release candidate build is tomorrow morning.
I'd prefer this get more bake time in nightly.

Flags: needinfo?(dmalyshau)
Attachment #9099661 - Flags: approval-mozilla-beta? → approval-mozilla-beta-

Comment on attachment 9100208 [details]
Force 256 alignment for texture uploads on AMD on macOS

Just a shade too last minute for my comfort levels as the RC build is tomorrow.

Attachment #9100208 - Flags: approval-mozilla-beta? → approval-mozilla-beta-

Liz, testing is very simple: run FF (edit: with WebRender enabled!) on macOS 10.15 with AMD GPU, the corruption in colors all over the screen would be obvious.

Flags: needinfo?(dmalyshau)

I would just like to report using webrender on macos looks like this:

https://pixelfed.social/storage/m/9415380b19c20b948c98d7c5f1fb65710b040a9e/06cd91a92d0439818e7e88631d135e73fbf12efd/u27KRHPMHaRkBtjDGSuonxhHBGSt4xDKE1zcQIdN.png

since macos 10.15 (Catalina), using an egpu with rx580

This is a vanilla profile with: user_pref("gfx.webrender.all", true); user_pref("gfx.webrender.enabled", true);

Webrender was working just fine for me in 10.14.

Our detection is barebone: let is_amd_macos = cfg!(target_os = "macos") && renderer_name.starts_with("AMD");, perhaps the rx580 returns a different format of the renderer string? 0011101000101001, do you have a glxinfo so that you could attache the output here?

Status: RESOLVED → REOPENED
Flags: needinfo?(0011101000101001)
Resolution: FIXED → ---

If you point me to some info on how to get this on a mac, I can try to get this.

Perhaps you could just attach the "about:support" page output.
I can see that on Windows rx580 is identified by the GL driver as "Radeon RX 580 Series", although it tells nothing about macOS drivers.

Attached file about-support.txt

Sorry for the delay.

Flags: needinfo?(0011101000101001)

Sorry, I should have noticed that earlier (thanks :mstange for the hint!) - you are running the Release Firefox 70.0. The fix is in Nightly, but it's going to take it a few more weeks before it reaches Beta/Release channels.

Status: REOPENED → RESOLVED
Closed: 5 years ago5 years ago
Resolution: --- → FIXED

I thought it was a good idea to report this but it seems I skimmed too much in this thread sorry. So January 7th in stable probably. Good to know thanks.

The nightly fix forks for me.

You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: