Shing Lyu [:shinglyu] Reporter
	Description • 8 years ago

The following reftests crashed stylo. 

To reproduce, run `./mach reftest --disable-e10s <filename>` or replace <filename> with `layout/reftest/reftest.list`.

The original log is also attached. 

==========================
TEST-UNEXPECTED-FAIL | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/reftest-sanity/no-root.html | application terminated with exit code 11
REFTEST PROCESS-CRASH | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/reftest-sanity/no-root.html | application crashed [None]
--
TEST-UNEXPECTED-FAIL | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/w3c-css/submitted/images3/object-fit-fill-png-001e.html | application terminated with exit code 11
REFTEST PROCESS-CRASH | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/w3c-css/submitted/images3/object-fit-fill-png-001e.html | application crashed [None]
--
TEST-UNEXPECTED-FAIL | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/w3c-css/received/css-values-3/calc-in-media-queries-001.html | application terminated with exit code 4
REFTEST PROCESS-CRASH | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/w3c-css/received/css-values-3/calc-in-media-queries-001.html | application crashed [None]
--
TEST-UNEXPECTED-FAIL | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/abs-pos/font-size-wrap.html | application terminated with exit code 11
REFTEST PROCESS-CRASH | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/abs-pos/font-size-wrap.html | application crashed [None]
--
TEST-UNEXPECTED-FAIL | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/backgrounds/background-clip-1.html | application terminated with exit code 11
REFTEST PROCESS-CRASH | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/backgrounds/background-clip-1.html | application crashed [None]
--
TEST-UNEXPECTED-FAIL | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/bidi/817406-2.html | application terminated with exit code 11
REFTEST PROCESS-CRASH | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/bidi/817406-2.html | application crashed [None]
--
TEST-UNEXPECTED-FAIL | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/border-radius/corner-3.html | application terminated with exit code 11
REFTEST PROCESS-CRASH | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/border-radius/corner-3.html | application crashed [None]
--
TEST-UNEXPECTED-FAIL | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/box-shadow/1178575.html | application terminated with exit code 11
REFTEST PROCESS-CRASH | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/box-shadow/1178575.html | application crashed [None]
--
TEST-UNEXPECTED-FAIL | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/css-animations/partially-out-of-view-animation.html | application terminated with exit code 11
REFTEST PROCESS-CRASH | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/css-animations/partially-out-of-view-animation.html | application crashed [None]
--
TEST-UNEXPECTED-FAIL | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/css-gradients/radial-shape-closest-corner-1a.html | application terminated with exit code 11
REFTEST PROCESS-CRASH | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/css-gradients/radial-shape-closest-corner-1a.html | application crashed [None]
--
TEST-UNEXPECTED-FAIL | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/text-overflow/selection.html | application terminated with exit code 11
REFTEST PROCESS-CRASH | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/text-overflow/selection.html | application crashed [None]
--
TEST-UNEXPECTED-FAIL | file:///home/shinglyu/workspace/stylo/gecko-dev/dom/tests/reftest/bug439965.html | application terminated with exit code 11
REFTEST PROCESS-CRASH | file:///home/shinglyu/workspace/stylo/gecko-dev/dom/tests/reftest/bug439965.html | application crashed [None]
--
TEST-UNEXPECTED-FAIL | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/dom/multipleappendwitheditable.xhtml | application terminated with exit code 11
REFTEST PROCESS-CRASH | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/dom/multipleappendwitheditable.xhtml | application crashed [None]
--
TEST-UNEXPECTED-FAIL | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/first-line/stress-3.html | application terminated with exit code 11
REFTEST PROCESS-CRASH | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/first-line/stress-3.html | application crashed [None]
--
TEST-UNEXPECTED-FAIL | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/font-face/cross-iframe-1.html | application terminated with exit code 11
REFTEST PROCESS-CRASH | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/font-face/cross-iframe-1.html | application crashed [None]
--
TEST-UNEXPECTED-FAIL | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/font-inflation/list-1.html | application terminated with exit code 11
REFTEST PROCESS-CRASH | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/font-inflation/list-1.html | application crashed [None]
--
TEST-UNEXPECTED-FAIL | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/font-loading-api/dynamic-insert-1.html | application terminated with exit code 11
REFTEST PROCESS-CRASH | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/font-loading-api/dynamic-insert-1.html | application crashed [None]
--
TEST-UNEXPECTED-FAIL | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/forms/fieldset/abs-pos-child-sizing.html | application terminated with exit code 11
REFTEST PROCESS-CRASH | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/forms/fieldset/abs-pos-child-sizing.html | application crashed [None]
--
TEST-UNEXPECTED-FAIL | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/image/image-object-fit-with-background-1.html | application terminated with exit code 11
REFTEST PROCESS-CRASH | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/image/image-object-fit-with-background-1.html | application crashed [None]
--
TEST-UNEXPECTED-FAIL | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/layers/move-to-background-1.html | application terminated with exit code 11
REFTEST PROCESS-CRASH | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/layers/move-to-background-1.html | application crashed [None]
--
TEST-UNEXPECTED-FAIL | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/mathml/dir-1.html | application terminated with exit code 11
REFTEST PROCESS-CRASH | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/mathml/dir-1.html | application crashed [None]
--
TEST-UNEXPECTED-FAIL | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/margin-collapsing/block-xhtml-root-1a.xhtml | application terminated with exit code 11
REFTEST PROCESS-CRASH | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/margin-collapsing/block-xhtml-root-1a.xhtml | application crashed [None]
--
TEST-UNEXPECTED-FAIL | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/object/image.html | application terminated with exit code 11
REFTEST PROCESS-CRASH | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/object/image.html | application crashed [None]
--
TEST-UNEXPECTED-FAIL | file:///home/shinglyu/workspace/stylo/gecko-dev/parser/htmlparser/tests/reftest/bug608373-1.html | application terminated with exit code 6
REFTEST PROCESS-CRASH | file:///home/shinglyu/workspace/stylo/gecko-dev/parser/htmlparser/tests/reftest/bug608373-1.html | application crashed [None]
--
TEST-UNEXPECTED-FAIL | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/pixel-rounding/background-color-base.html | application terminated with exit code 11
REFTEST PROCESS-CRASH | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/pixel-rounding/background-color-base.html | application crashed [None]
--
TEST-UNEXPECTED-FAIL | file:///home/shinglyu/workspace/stylo/gecko-dev/dom/plugins/test/reftest/plugin-background.html | application terminated with exit code 11
REFTEST PROCESS-CRASH | file:///home/shinglyu/workspace/stylo/gecko-dev/dom/plugins/test/reftest/plugin-background.html | application crashed [None]
--
TEST-UNEXPECTED-FAIL | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/position-dynamic-changes/multiple-changes.html | application terminated with exit code 11
REFTEST PROCESS-CRASH | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/position-dynamic-changes/multiple-changes.html | application crashed [None]
--
TEST-UNEXPECTED-FAIL | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/pagination/abspos-breaking-003.html | application terminated with exit code 11
REFTEST PROCESS-CRASH | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/pagination/abspos-breaking-003.html | application crashed [None]
--
TEST-UNEXPECTED-FAIL | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/selection/pseudo-element-of-native-anonymous.html | application terminated with exit code 11
REFTEST PROCESS-CRASH | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/selection/pseudo-element-of-native-anonymous.html | application crashed [None]
--
TEST-UNEXPECTED-FAIL | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/svg/smil/container/deferred-anim-1.xhtml | application terminated with exit code 11
REFTEST PROCESS-CRASH | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/svg/smil/container/deferred-anim-1.xhtml | application crashed [None]
--
TEST-UNEXPECTED-FAIL | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/table-anonymous-boxes/162063-1.xhtml | application terminated with exit code 11
REFTEST PROCESS-CRASH | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/table-anonymous-boxes/162063-1.xhtml | application crashed [None]
--
TEST-UNEXPECTED-FAIL | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/table-bordercollapse/bc_borderoffset1.html | application terminated with exit code 11
REFTEST PROCESS-CRASH | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/table-bordercollapse/bc_borderoffset1.html | application crashed [None]
--
TEST-UNEXPECTED-FAIL | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/table-width/balancing-1.html | application terminated with exit code 6
REFTEST PROCESS-CRASH | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/table-width/balancing-1.html | application crashed [None]
--
TEST-UNEXPECTED-FAIL | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/text-shadow/text-shadow-selected-1.html | application terminated with exit code 11
REFTEST PROCESS-CRASH | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/text-shadow/text-shadow-selected-1.html | application crashed [None]
--
TEST-UNEXPECTED-FAIL | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/transform-3d/backface-visibility-3.html | application terminated with exit code 11
REFTEST PROCESS-CRASH | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/transform-3d/backface-visibility-3.html | application crashed [None]
--
TEST-UNEXPECTED-FAIL | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/web-animations/stacking-context-transform-changing-display-property.html | application terminated with exit code 11
REFTEST PROCESS-CRASH | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/web-animations/stacking-context-transform-changing-display-property.html | application crashed [None]
--
TEST-UNEXPECTED-FAIL | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/webcomponents/input-transition-1.html | application terminated with exit code 11
REFTEST PROCESS-CRASH | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/webcomponents/input-transition-1.html | application crashed [None]
--
TEST-UNEXPECTED-FAIL | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/xul/grid/reftests/not-full-basic.xul | application terminated with exit code 11
REFTEST PROCESS-CRASH | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/xul/grid/reftests/not-full-basic.xul | application crashed [None]
--
TEST-UNEXPECTED-FAIL | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/writing-mode/1106669-1-intrinsic-for-container.html | application terminated with exit code 11
REFTEST PROCESS-CRASH | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/writing-mode/1106669-1-intrinsic-for-container.html | application crashed [None]
--
TEST-UNEXPECTED-FAIL | file:///home/shinglyu/workspace/stylo/gecko-dev/dom/html/reftests/468263-1c.html | application terminated with exit code 11
REFTEST PROCESS-CRASH | file:///home/shinglyu/workspace/stylo/gecko-dev/dom/html/reftests/468263-1c.html | application crashed [None]
--
TEST-UNEXPECTED-FAIL | file:///home/shinglyu/workspace/stylo/gecko-dev/editor/reftests/caret_on_positioned.html | application terminated with exit code 11
REFTEST PROCESS-CRASH | file:///home/shinglyu/workspace/stylo/gecko-dev/editor/reftests/caret_on_positioned.html | application crashed [None]
===============================

	Shing Lyu [:shinglyu] Reporter
	Comment 1 • 8 years ago

Chris, can you help find someone who is more experienced in Gecko to look into this?

	Chris Peterson [:cpeterson]
	Comment 2 • 8 years ago

(In reply to Shing Lyu [:shinglyu] from comment #1)
> Chris, can you help find someone who is more experienced in Gecko to look
> into this?

Sure. Are these crashes in the Stylo/Gecko C++ glue or panics in Stylo's Rust code?

	Bobby Holley (:bholley)
	Comment 3 • 8 years ago

I'm digging through these.

	Bobby Holley (:bholley)
	Comment 4 • 8 years ago

I've gone through and fixed various issues here, and categorized the remaining ones. Once the current front-of-queue lands on servo/servo and I push the updated gecko-dev branch, these should be the remaining failures:

TEST-UNEXPECTED-FAIL | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/w3c-css/received/css-values-3/calc-in-media-queries-001.html | application terminated with exit code 4
REFTEST PROCESS-CRASH | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/w3c-css/received/css-values-3/calc-in-media-queries-001.html | application crashed [None]
--
hit unreachable!() during media query parsing: https://github.com/servo/servo/blob/cbf71a2cf39e792f75309e68f7cabe862d4a70eb/components/style/media_queries.rs#L44


TEST-UNEXPECTED-FAIL | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/css-animations/partially-out-of-view-animation.html | application terminated with exit code 11
REFTEST PROCESS-CRASH | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/css-animations/partially-out-of-view-animation.html | application crashed [None]
--
Hangs, presumably because the test waits on a "animationend" event that never arrives.


TEST-UNEXPECTED-FAIL | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/font-inflation/list-1.html | application terminated with exit code 11
REFTEST PROCESS-CRASH | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/font-inflation/list-1.html | application crashed [None]
--
We call mozilla::ServoStyleSet::AddDocStyleSheet with a ServoStyleSheet for whom RawSheet() returns null.



TEST-UNEXPECTED-FAIL | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/margin-collapsing/block-xhtml-root-1a.xhtml | application terminated with exit code 11
REFTEST PROCESS-CRASH | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/margin-collapsing/block-xhtml-root-1a.xhtml | application crashed [None]
--
Assertion failure: aNameSpaceID > 0 && (int64_t) aNameSpaceID <= (int64_t) mURIArray.Length(), at /files/mozilla/stylo/bb/gecko/dom/base/nsNameSpaceManager.h:43
ç      \#01: nsNameSpaceManager::NameSpaceURIAtom(int) (nsNameSpaceManager.h:43, in XUL)
#02: Gecko_Namespace (ServoBindings.cpp:158, in XUL)
#03: _$LT$wrapper..GeckoElement$LT$$u27$le$GT$$u20$as$u20$selectors..Element$GT$::get_namespace::h7a2f0ddcbd2acca7 (wrapper.rs:468, in XUL)




TEST-UNEXPECTED-FAIL | file:///home/shinglyu/workspace/stylo/gecko-dev/parser/htmlparser/tests/reftest/bug608373-1.html | application terminated with exit code 6
REFTEST PROCESS-CRASH | file:///home/shinglyu/workspace/stylo/gecko-dev/parser/htmlparser/tests/reftest/bug608373-1.html | application crashed [None]
--
Crashes during the cascade because the <html> element (the parent of the element we're attempt to style) has not been styled for some reason.



TEST-UNEXPECTED-FAIL | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/pagination/abspos-breaking-003.html | application terminated with exit code 11
REFTEST PROCESS-CRASH | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/pagination/abspos-breaking-003.html | application crashed [None]
--
Assertion failure: aNode->IsDirtyForServo() || aNode->HasDirtyDescendantsForServo(), at /files/mozilla/stylo/bb/gecko/layout/style/ServoStyleSet.cpp:421




TEST-UNEXPECTED-FAIL | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/selection/pseudo-element-of-native-anonymous.html | application terminated with exit code 11
REFTEST PROCESS-CRASH | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/selection/pseudo-element-of-native-anonymous.html | application crashed [None]
--
Assertion failure: mSource.IsGeckoRuleNode(), at /files/mozilla/stylo/bb/gecko/layout/style/nsStyleContext.h:325


TEST-UNEXPECTED-FAIL | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/svg/smil/container/deferred-anim-1.xhtml | application terminated with exit code 11
REFTEST PROCESS-CRASH | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/svg/smil/container/deferred-anim-1.xhtml | application crashed [None]
Assertion failure: aStyleContext->PresContext()->StyleSet()->IsGecko() (ServoStyleSet should not use StyleAnimationValue for animations), at /files/mozilla/stylo/bb/gecko/layout/style/StyleAnimationValue.cpp:2831


--
TEST-UNEXPECTED-FAIL | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/web-animations/stacking-context-transform-changing-display-property.html | application terminated with exit code 11
REFTEST PROCESS-CRASH | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/web-animations/stacking-context-transform-changing-display-property.html | application crashed [None]
--
Assertion failure: aStyleContext->PresContext()->StyleSet()->IsGecko() (ServoStyleSet should not use StyleAnimationValue for animations), at /files/mozilla/stylo/bb/gecko/layout/style/StyleAnimationValue.cpp:2831



TEST-UNEXPECTED-FAIL | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/webcomponents/input-transition-1.html | application terminated with exit code 11
REFTEST PROCESS-CRASH | file:///home/shinglyu/workspace/stylo/gecko-dev/layout/reftests/webcomponents/input-transition-1.html | application crashed [None]
--
Assertion failure: sheet->IsGecko() (GatherRuleProcessor must only be called for nsXBLPrototypeResources objects with Gecko-flavored style backends), at /files/mozilla/stylo/bb/gecko/dom/xbl/nsXBLPrototypeResources.cpp:154

	Bobby Holley (:bholley)
	Comment 5 • 8 years ago

The first one looks like an easy servo bug. We don't have media queries hooked up yet, but we shouldn't crash the way we appear to be here. Shing, do you want to take a look at it?

	Chris Peterson [:cpeterson]
	Comment 6 • 8 years ago

(In reply to Bobby Holley (:bholley) (busy with Stylo) from comment #4)
> I've gone through and fixed various issues here, and categorized the
> remaining ones. Once the current front-of-queue lands on servo/servo and I
> push the updated gecko-dev branch, these should be the remaining failures:

Bobby, should we file separate bugs for the remaining issues so we can delegate them to more people?

	Bobby Holley (:bholley)
	Comment 7 • 8 years ago

(In reply to Chris Peterson [:cpeterson] from comment #6)
> (In reply to Bobby Holley (:bholley) (busy with Stylo) from comment #4)
> > I've gone through and fixed various issues here, and categorized the
> > remaining ones. Once the current front-of-queue lands on servo/servo and I
> > push the updated gecko-dev branch, these should be the remaining failures:
> 
> Bobby, should we file separate bugs for the remaining issues so we can
> delegate them to more people?

That sounds good, yeah. Do you have the cycles to file them? Most useful to file with the assertion failure or description as the summary and mention the reftest in comment 0.

	Chris Peterson [:cpeterson]
	Comment 8 • 8 years ago

I filed bugs for the crashes and assertion failures list in comment 4.

	Bobby Holley (:bholley)
	Comment 9 • 8 years ago

Thanks!

	Shing Lyu [:shinglyu] Reporter
	Comment 10 • 7 years ago

Manish, here are the crashes if you are interested.

	Shing Lyu [:shinglyu] Reporter
	Comment 11 • 7 years ago

Some more crashes from the latest reftest run.

	Cameron McCormack (:heycam)
	Comment 12 • 7 years ago

Thanks, looks like not too many.  Could you see if any of these crashes are not covered by the dependencies of this bug, and file new ones if they aren't?

	Shing Lyu [:shinglyu] Reporter
	Comment 13 • 7 years ago

Done!

Except for the 3 bugs I just filed, there are quite a few in bug 1323649 and bug 1323665

	Shing Lyu [:shinglyu] Reporter
	Comment 14 • 7 years ago

I hacked the reftest to make it run stylo-to-gecko reftest using the latest reftest.list (including all the newly added tests, and use Gecko's skip-if/fail-if flags), here are the summary:

Summary
------------------------
crashes:          6
unexpected-fail:  5074
unexpected-pass:  248
known-fail:       182
known-skip:       251
passes:           9455
------------------------
total:            15208
------------------------

The only crashes left are:

layout/reftests/invalidation/clip-path-invalidation-1a.html 
layout/reftests/invalidation/clip-path-invalidation-1b.html 
layout/reftests/w3c-css/received/selectors-4/focus-within-shadow-002.html 
layout/reftests/w3c-css/received/selectors-4/focus-within-shadow-004.html 
layout/reftests/w3c-css/received/selectors-4/of-type-selectors.xhtml 
layout/reftests/webcomponents/input-transition-1.html 

That's too good to be true, so I'll double check if my modified reftest runner are actually working or not :/
The stylo build is commit 08d398913, on linux-64

	Shing Lyu [:shinglyu] Reporter
	Comment 15 • 7 years ago

Also the number suggest we'll need to add ~5000 `fail-if(stylo)` annotations to the reftest.list, which I believe is still too high. The top 5 fails-if only have this much:

    155 fails-if(Android)
    127 fails-if(webrender)
     17 fails-if(!haveTestPlugin)
     16 fails-if(winWidget&&layersGPUAccelerated&&!d3d11)
     12 fails-if(layersGPUAccelerated)

And skip-if only have 

    237 skip-if(Android)
     68 skip-if(!asyncPan)
     61 skip-if(!Android)
     21 skip-if(!webrender)
     15 skip-if(!cocoaWidget)

I gut feeling is that we need to reduce that number to less then 1000 or even lower.

Bobby, do you have some opinion on this?

	Shing Lyu [:shinglyu] Reporter
	Comment 16 • 7 years ago

The detailed log for the reftest run can be found here: https://drive.google.com/open?id=0B5hP4LvYzmF5ZE10cm9rX0liVms

	Bobby Holley (:bholley)
	Comment 17 • 7 years ago

(In reply to Shing Lyu [:shinglyu] from comment #14)
> That's too good to be true, so I'll double check if my modified reftest
> runner are actually working or not :/
> The stylo build is commit 08d398913, on linux-64

This is probably because most of our crashes are in the form of assertions, and from the logs I think you're running a build with --disable-debug. I think you should probably test with --enable-debug --enable-optimize.

I don't think the manifest churn itself is necessarily a problem. However, if we're only passing ~70% of the total tests, there's a decent chance that any newly-added tests will also need fails-if(stylo), which is a bit annoying for other layout developers (since they won't find out until try or inbound).

The primary disadvantage of our current setup is that we don't notice when tests stop crashing (since there's no equivalent of TEST-UNEXPECTED-PASS for a disabled test). How easy is it to automatically retest the tests we have disabled in our stylo manifests and re-enable the passing ones?

	Shing Lyu [:shinglyu] Reporter
	Comment 18 • 7 years ago

@Bobby:

I'm working on another patch for reftest that enabled it to recover from crashes [1]. This way we don't need to disable them, we can simply run all the tests every time. We might also need a new flag "crash-if(stylo)" which can give us TEST-UNEXPECTED-PASS if it stops crashing.

Does that sound like a good plan to you?

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1344991

	Bobby Holley (:bholley)
	Comment 19 • 7 years ago

(In reply to Shing Lyu [:shinglyu] from comment #18)
> @Bobby:
> 
> I'm working on another patch for reftest that enabled it to recover from
> crashes [1]. This way we don't need to disable them, we can simply run all
> the tests every time. We might also need a new flag "crash-if(stylo)" which
> can give us TEST-UNEXPECTED-PASS if it stops crashing.
> 
> Does that sound like a good plan to you?
> 
> [1] https://bugzilla.mozilla.org/show_bug.cgi?id=1344991

I think it will mostly depend on performance, i.e. how quickly the content process can be restarted in --enable-optimize builds (the numbers might differ with and without --enable-debug). If crashing and spinning up a new content process is many times slower than a normal page load, then our high crash rate might cause the reftest jobs to take a lot longer, which might not be acceptable.

If the performance is good enough though, this sounds like a great plan, and a great contribution to developer productivity even beyond stylo. Just need to measure.

	Shing Lyu [:shinglyu] Reporter
	Comment 20 • 7 years ago

I just re-run the test using the original reftest.list with my modified reftest runner: https://drive.google.com/open?id=0B5hP4LvYzmF5YWVMU0t3MExaT2c

Here are the summary:

Summary
------------------------
crashes:          25
unexpected-fail:  4908
unexpected-pass:  18
known-fail:       423
known-skip:       251
passes:           9837
------------------------
total:            15217
------------------------

	Shing Lyu [:shinglyu] Reporter
	Comment 21 • 7 years ago

(In reply to Shing Lyu [:shinglyu] from comment #20)
> I just re-run the test using the original reftest.list with my modified
> reftest runner: https://drive.google.com/open?id=0B5hP4LvYzmF5YWVMU0t3MExaT2c

That's a debug build, for getting the assertion crashes

	Bobby Holley (:bholley)
	Comment 22 • 7 years ago

That's great! At least 25 crashes is a manageable number.

	David Bruant
	Comment 23 • 6 years ago

There are no dependent bugs and there doesn't seem to be having for a year
Maybe this bug can be closed and crashes taken care of independently as they arrive

	Emilio Cobos Álvarez (:emilio)
	Comment 24 • 6 years ago

I agree.