Fractional scaling currently comes at a high performance cost as we only support integer scaling for rendering. The overdraw ratio is something like (ceil(scale) / scale)^2, which for 125% scaling means we're drawing 2.56 times as much as we should.

In theory we already support everything needed to have proper fractional scaling: Webrender support appears to be in great shape and on the Wayland side we can use the wp_viewporter protocol to have arbitrary buffer size <-> surface size relationships. The main blocker remains the lack of proper negotiation between client and compositor about the optimal buffer size.

In order to speed up the upstream discussion, lets implement it via a fixed value that can be set in about:config for testing purposes. It will, of course, require wp_viewporter support from the compositor.

Fractional scaling currently comes at a high performance cost as we
only support integer scaling for rendering. The overdraw ratio is
something like (ceil(scale) / scale)^2, which for 125% scaling means
we're drawing 2.56 times as much as we should.

In theory we already support everything needed to have proper fractional
scaling: Webrender support appears to be in great shape and on the Wayland
side we can use the wp_viewporter protocol to have arbitrary
buffer size <-> surface size relationships. The main blocker remains the
lack of proper negotiation between client and compositor about the
optimal buffer size.

In order to speed up the upstream discussion, lets implement it via a
fixed value that can be set in about:config for testing purposes
(widget.wayland.fractional_buffer_scale).
It will, of course, require wp_viewporter support from the compositor.

https://hg.mozilla.org/mozilla-central/rev/3b393f95a2b4

I’m trying this out at present after being mildly displeased with devicePixelRatio being 2 rather than 1.5; how interested are you in bug reports? I’ve already filed bug 1730114, but as I collect more I’m reflecting that I’m not sure if you want this stuff yet, or if the upstream negotiation should happen first, and it’s distinctly possible that all of this stuff has the same root cause anyway. And at least under Sway, most of this stuff can’t be reproduced on master gtk3 anyway, needing gtk!3898 to fix popups.

(“More”: checkboxes in menus are overlarge; popups are overlarge for the first frame or so; popups sometimes render with their bottoms clipped off—keep clicking the Application Menu button to reproduce that; or altogether invisible until sufficient interaction—seen most of the time on extension toolbar popups; menus two or three levels deep are very overlarge and flicker persistently; context menu occasionally gets placed incorrectly.)

(In reply to Chris Morgan from comment #4)

how interested are you in bug reports? I’ve already filed bug 1730114, but as I collect more I’m reflecting that I’m not sure if you want this stuff yet, or if the upstream negotiation should happen first, and it’s distinctly possible that all of this stuff has the same root cause anyway.

Sorry for the late reply! I won't work much on this for now until the upstream negotiations are settled - help with that is strongly appreciated though ;)

The proposed protocol in https://gitlab.freedesktop.org/wayland/wayland-protocols/-/merge_requests/143 would allow us to obtain the correct scaling value (instead of hardcoding things) while using the already implemented wp_viewporter solution to do the scaling. So implementing it should be easy on top of the existing work.

The test clients in https://gitlab.freedesktop.org/kennylevinsen/fractional-scaling should help compositors get things right - and can serve as a debug/comparison tool in case we run into issues with Firefox.