Andreas Tolfsen ❲:ato❳ Reporter
	Description • 6 years ago

wptrunner provides meta data that lets you skip subtests.  This does
not currently work for wdspec tests because we don’t feed this
data to pytest.

According to davehunt it is possible to tell pytest which tests to
run and which not to by instrumenting the test discovery/ingestion.
I don’t have the details at hand.

	Henrik Skupin [:whimboo][⌚️UTC+1]
	Comment 1 • 6 years ago

Dave or Raphael, can one of you send us some details in how to instruct pytest about which tests to run and which not based on our manifest file wptrunner makes use of?

	Henrik Skupin [:whimboo][⌚️UTC+1]
	Comment 2 • 6 years ago

Maybe we can use one of the collection hooks to prevent running a specific test:

https://docs.pytest.org/en/documentation-restructure/how-to/writing_plugins.html#collection-hooks

	Raphael Aurich [:raphael] UTC+02:00
	Comment 3 • 6 years ago

Do we have the means to get a list of test items that we want to run and which ones to skip? The `pytest_collection_modifyitems` hook is great for these kinds of scenarios. I used it in my blogpost below, in which I demonstrated how to select tests based on whether they use a particular fixture.

Hope this helps!


http://hackebrot.github.io/pytest-tricks/run_tests_using_fixture/

	Henrik Skupin [:whimboo][⌚️UTC+1]
	Comment 4 • 6 years ago

So I think what has to be done here is:

1) Make sure that the manifest parser correctly identifies the disabled keyword for sub tests. Looks like this doesn't really work, or is something which `--list-disabled` doesn't support? Example here: 2d.pattern.image.broken.html.

2) Ensure we can pass-through the disabled status of sub tests to the pytestrunner module

3) Having a hook which checks each of the collected sub tests for disabled status, and marks them as skipped for pytest

James, could you give some info for 1) Is that expected, or just a missing feature in --list-disabled? Given the usage of the disabled keyword for subtests in other wpt test suites, it should work.

	James Graham [:jgraham]
	Comment 5 • 6 years ago

I'm pretty sure that --list-disabled doesn't support that. Following through from a search for list_disabled in SF, we see https://searchfox.org/mozilla-central/source/testing/web-platform/tests/tools/wptrunner/wptrunner/testloader.py#559 where it's clear that it's just generating a list of top level tests that are disabled.

	Henrik Skupin [:whimboo][⌚️UTC+1]
	Comment 6 • 6 years ago

(In reply to James Graham [:jgraham] from comment #5)
> I'm pretty sure that --list-disabled doesn't support that. 

But does that mean that it is not wanted, or just hasn't been implemented yet because no-one needed it. Having it available for sub-tests would be kinda helpful, maybe via an option to the command.

	James Graham [:jgraham]
	Comment 7 • 6 years ago

I don't think anyone spescifically depends on that functionaility. Adding more features seems fine, but it's generally not possible to tell if a subtest actually exists or not (without running the tests), so you might end up listing some tests that were later deleted.