Open
Bug 1427990
Opened 6 years ago
Updated 2 years ago
Disabled status for subtests in web-platform-tests is not obeyed
Categories
(Testing :: web-platform-tests, defect, P3)
Tracking
(Not tracked)
NEW
People
(Reporter: whimboo, Unassigned)
References
Details
I noticed that behavior over on bug 1403923 by trying to disable a subtest like: > [user_prompts.py::test_handle_prompt_twice] > disabled: https://bugzilla.mozilla.org/show_bug.cgi?id=1425559 This disabled flag is not taking into account, and instead the test is run all the time. We actually create ManifestItems for each subtest but then maybe miss to read or evaluate the data. We have a lot of others marked as disabled (at least in wdspec), and all of them are getting run anyway. As such the failure on bug 1407383 is still visible a lot for jobs on Linux. James, can you please have a look?
Flags: needinfo?(james)
Reporter | ||
Comment 1•6 years ago
|
||
Please drop the part for bug 1407383. This is not a subtest, but `navigation/current_url.py.ini` is.
Comment 2•6 years ago
|
||
For subtests the general strategy is to run the full test file and ignore the results of subtests. Obviously this doesn't work if they affect the execution of later tests. However in general this is the only strategy that can work since we don't have any way to selectively not execute parts of a testharness.js test. For wdspec (i.e. pytest-based) tests we could do better by hacking the pytest loader, but that hasn't been a priority. If this is not even ignoring the test result that seems more like a bug, but I'd need to see an actual job to check that yu have the correct subtest name &c.
Flags: needinfo?(james)
Reporter | ||
Comment 3•6 years ago
|
||
(In reply to James Graham [:jgraham] from comment #2) > If this is not even ignoring the test result that seems more like a bug, but > I'd need to see an actual job to check that yu have the correct subtest name > &c. No, the test result itself was obeyed correctly, but the test itself caused a shutdown hang, which other tests in that file would be affected off due to a timeout happened at this point. What I was surprised is that we cannot set a TIMEOUT result for a subtest build only for a test. This brought some extra confusion in here. It would be great to be able to disable specific subtests via pytest. Maybe Dave knows a way how to do it without too much work involved.
Flags: needinfo?(dave.hunt)
Comment 4•6 years ago
|
||
> It would be great to be able to disable specific subtests via pytest. Maybe Dave knows a way how to do it without too much work involved. You can filter out tests with pytest by annotating them with markers and specifying a filter. I'm not sure how this applies to subtests and the way we're executing pytest though. You can read details on using markers at https://docs.pytest.org/en/latest/example/markers.html
Flags: needinfo?(dave.hunt)
Comment 5•6 years ago
|
||
(In reply to Henrik Skupin (:whimboo) from comment #3) > It would be great to be able to disable specific subtests via > pytest. Maybe Dave knows a way how to do it without too much work > involved. One very quick Google search shows you pass the "-k 'not subtest_name'" [1] flag to skip the running of a pytest (or child test in WPT parlance). [1] https://docs.pytest.org/en/latest/example/markers.html
Comment 6•6 years ago
|
||
davehunt further mentioned that -k does a bit of fuzzy matching which is not necessarily what we want if we know the subtest names upfront. He mentioned we could perhaps do something at test collection using a hook: https://docs.pytest.org/en/latest/writing_plugins.html#_pytest.hookspec.pytest_collection_modifyitems
Updated•2 years ago
|
Severity: normal → S3
You need to log in
before you can comment on or make changes to this bug.
Description
•