Open Bug 1427990 Opened 6 years ago Updated 2 years ago

Disabled status for subtests in web-platform-tests is not obeyed

Categories

(Testing :: web-platform-tests, defect, P3)

Version 3
defect

Tracking

(Not tracked)

People

(Reporter: whimboo, Unassigned)

References

Details

I noticed that behavior over on bug 1403923 by trying to disable a subtest like:

>  [user_prompts.py::test_handle_prompt_twice]
>    disabled: https://bugzilla.mozilla.org/show_bug.cgi?id=1425559

This disabled flag is not taking into account, and instead the test is run all the time. 

We actually create ManifestItems for each subtest but then maybe miss to read or evaluate the data.

We have a lot of others marked as disabled (at least in wdspec), and all of them are getting run anyway. As such the failure on bug 1407383 is still visible a lot for jobs on Linux.

James, can you please have a look?
Flags: needinfo?(james)
Please drop the part for bug 1407383. This is not a subtest, but `navigation/current_url.py.ini` is.
For subtests the general strategy is to run the full test file and ignore the results of subtests. Obviously this doesn't work if they affect the execution of later tests. However in general this is the only strategy that can work since we don't have any way to selectively not execute parts of a testharness.js test. For wdspec (i.e. pytest-based) tests we could do better by hacking the pytest loader, but that hasn't been a priority.

If this is not even ignoring the test result that seems more like a bug, but I'd need to see an actual job to check that yu have the correct subtest name &c.
Flags: needinfo?(james)
(In reply to James Graham [:jgraham] from comment #2)
> If this is not even ignoring the test result that seems more like a bug, but
> I'd need to see an actual job to check that yu have the correct subtest name
> &c.

No, the test result itself was obeyed correctly, but the test itself caused a shutdown hang, which other tests in that file would be affected off due to a timeout happened at this point. What I was surprised is that we cannot set a TIMEOUT result for a subtest build only for a test. This brought some extra confusion in here.

It would be great to be able to disable specific subtests via pytest. Maybe Dave knows a way how to do it without too much work involved.
Flags: needinfo?(dave.hunt)
> It would be great to be able to disable specific subtests via pytest. Maybe Dave knows a way how to do it without too much work involved.

You can filter out tests with pytest by annotating them with markers and specifying a filter. I'm not sure how this applies to subtests and the way we're executing pytest though. You can read details on using markers at https://docs.pytest.org/en/latest/example/markers.html
Flags: needinfo?(dave.hunt)
(In reply to Henrik Skupin (:whimboo) from comment #3)

> It would be great to be able to disable specific subtests via
> pytest. Maybe Dave knows a way how to do it without too much work
> involved.

One very quick Google search shows you pass the "-k 'not
subtest_name'" [1] flag to skip the running of a pytest (or child test
in WPT parlance).

  [1] https://docs.pytest.org/en/latest/example/markers.html
davehunt further mentioned that -k does a bit of fuzzy
matching which is not necessarily what we want if we
know the subtest names upfront.  He mentioned we could
perhaps do something at test collection using a hook:

	https://docs.pytest.org/en/latest/writing_plugins.html#_pytest.hookspec.pytest_collection_modifyitems
Blocks: 1431058
No longer blocks: 1431058
Moving to p3 because no activity for at least 24 weeks.
Priority: P1 → P3
Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.