Shing Lyu [:shinglyu] Assignee
	Description • 7 years ago

Bug 1344132 enable us to run stylo-to-gecko test using reftest.list. But it is not enabled on the try server. So we need to mark stylo-specific fails inside reftest.list, remove all the reftest-stylo.list and change the try configuration to run reftest.list instead of reftest-stylo.list.

	Shing Lyu [:shinglyu] Assignee
	Comment 4 • 7 years ago

The reftest-stylo.list thing is creating too much confusion, so I guess it's time to do this.

These patches depends on Bug 1344132. I'll land the two together when they both received r+.

There are some rules I use to prepare the patch:

* Things that clearly fail on stylo-vs-gecko test are marked as "fails-if(stylo)"
* Stylo-specific crashes or intermittents are marked as "skip-if(stylo)", we'll try them periodically to see if they are fixed or not.
* Things that are marked as "fails" or "fails-if(foo)" but is passing in stylo-vs-gecko test will be changed to "fails-if(!stylo)" and "fails-if(foo&&!stylo)"
* Some tests (e.g. reftest-sanity) uses the same test file against different ref files. We only test the first one and mark the rest as "skip-if(stylo)"
* We put the "skip-if(stylo)" and "fails-if(stylo)" at the end of the list, so it will override other conditions
* Tests in the form "foo.html#id" will break the test because it only scrolls but not reload. This is tricker to fix so I'll create a follow-up bug for that, there are only ~6 of them in the whole reftest, so we won't lose a lot of test coverage

	Shing Lyu [:shinglyu] Assignee
	Comment 5 • 7 years ago

Comment on attachment 8859004 [details]
Bug 1351548 - Add stylo-vs-gecko expectations to reftest.lists

Hi Bobby,

I usually ask heycam for the reftest stuff, but he is on PTO so would you mind review it for me?

	Bobby Holley (:bholley)
	Comment 6 • 7 years ago
mozreview-review

Comment on attachment 8859004 [details]
Bug 1351548 - Add stylo-vs-gecko expectations to reftest.lists

https://reviewboard.mozilla.org/r/131024/#review133998

::: layout/reftests/font-face/reftest.list:72
(Diff revision 1)
>  HTTP(..) == unicoderange-4.html unicoderange-4-ref.html
>  
>  # Dynamic changes
>  # we need to skip these because of the bug that's causing order-2.html to fail
> -HTTP(..) == enable-sheet-1.html enable-sheet-1-ref.html
> -skip HTTP(..) == enable-sheet-2.html multiple-in-family-1-ref.html
> +skip-if(stylo) HTTP(..) == enable-sheet-1.html enable-sheet-1-ref.html
> +skip skip-if(stylo) HTTP(..) == enable-sheet-2.html multiple-in-family-1-ref.html

This is kind of redundant I suppose, but we can fix it up later.

	Bobby Holley (:bholley)
	Comment 7 • 7 years ago
mozreview-review

Comment on attachment 8859005 [details]
Bug 1351548 - Switch from reftest-stylo.list to reftest.list on

https://reviewboard.mozilla.org/r/131026/#review134004

	Bobby Holley (:bholley)
	Comment 8 • 7 years ago
mozreview-review

Comment on attachment 8859006 [details]
Bug 1351548 - Remove reftest-stylo.lists

https://reviewboard.mozilla.org/r/131028/#review134006

	Bobby Holley (:bholley)
	Comment 9 • 7 years ago

Nice! I didn't review every line of the diffs but this seems to be doing the right thing, and CI will give us the canonical results anyway. Let's get this landed - it's one step closer to victory!

	Pulsebot
	Comment 10 • 7 years ago

Pushed by slyu@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/1f9d0f8e65b9
Add stylo-vs-gecko expectations to reftest.lists.  r=bholley
https://hg.mozilla.org/integration/mozilla-inbound/rev/4d8dfee7ebf3
Switch from reftest-stylo.list to reftest.list on linux64-stylo. r=bholley
https://hg.mozilla.org/integration/mozilla-inbound/rev/aebb0ceeb20a
Remove reftest-stylo.lists. r=bholley

	Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)
	Comment 11 • 7 years ago

Backed out bug 1344132 and bug 1351548 for failing reftest reftest-opaque-layer-fail.html and many reftest failures for Stylo builds:

Bug 1344132
https://hg.mozilla.org/integration/mozilla-inbound/rev/1d3a3ae57b06ae9ef71cf670727ee099aa1e248c

Bug 1351548
https://hg.mozilla.org/integration/mozilla-inbound/rev/6232431e65ef49c045f822c4c1c1335f9ff6d2a9
https://hg.mozilla.org/integration/mozilla-inbound/rev/f75640e4a0cc3deb4cc7d7b1d1aa544916ad5c60
https://hg.mozilla.org/integration/mozilla-inbound/rev/e4b8f7c680ad1d88d2eeefb2ad63c656b435fc8e

https://hg.mozilla.org/integration/mozilla-inbound/rev/1f9d0f8e65b91a5fee031950fb3940be44381194#l142.144 has
> fails fails-if(stylo)
As far as I know, the "fails-if" will overwrite the "fails".

For stylo, there are more assertions than expected. Please also check the other reftest failures:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=aebb0ceeb20ae6031f456f5893db0ba4ec96d9e9&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable

	Pulsebot
	Comment 12 • 7 years ago

Pushed by slyu@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/46fc8d459483
Add stylo-vs-gecko expectations to reftest.lists.  r=bholley
https://hg.mozilla.org/integration/mozilla-inbound/rev/3940a4760c84
Switch from reftest-stylo.list to reftest.list on linux64-stylo. r=bholley
https://hg.mozilla.org/integration/mozilla-inbound/rev/e1ceacb7312d
Remove reftest-stylo.lists. r=bholley

	Pulsebot
	Comment 13 • 7 years ago

Pushed by slyu@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/107d824d3657
Update stylo reftest expectations. r=me

	Carsten Book [:Tomcat]
	Comment 14 • 7 years ago
bugherder

https://hg.mozilla.org/mozilla-central/rev/46fc8d459483
https://hg.mozilla.org/mozilla-central/rev/3940a4760c84
https://hg.mozilla.org/mozilla-central/rev/e1ceacb7312d
https://hg.mozilla.org/mozilla-central/rev/107d824d3657

	Pulsebot
	Comment 15 • 7 years ago

Pushed by kgupta@mozilla.com:
https://hg.mozilla.org/mozilla-central/rev/1c6fa11d1fed
Follow-up to fix incorrect changes in cset 46fc8d459483. r=me a=Tomcat

	Kartikaya Gupta (email:kats@mozilla.staktrace.com)
	Comment 16 • 7 years ago

FYI ^ there were some errors when merging the reftest.list files. Two of the four errors were showing up as R4 failures on m-c because of TEST-UNEXPECTED-PASS. The other two were fuzzy passes so they are green regardless of whether they are marked fails-if or not (we really should fix that...). You might want to recheck that cset to make sure there weren't other errors, I only looked for webrender mismatches.

	Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)
	Comment 17 • 7 years ago

This accidentally re-enabled the test disabled in bug 1351548. Please check that no other changes to test expectations got reverted by accident.

	Masatoshi Kimura [:emk]
	Comment 18 • 7 years ago

This bug is 1351548.

	Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)
	Comment 19 • 7 years ago

Thank you. The test that got re-enabled is bug 1352942.