{{ message }}
Repository navigation
Is there a way to mark xfail tests, also with matching output? #129
Description
I would like to be able:
- Test is marked as
xfailif-f it does not matchout: |, but matchesxfail: |(imaginary attribute), but continues - Test is marked as
xpassif-f it matchesout: |(preferably strict, since it will lead to actually noticing in CI/CD - Otherwise it is marked as
fail
This is clearly wrong:
https://github.com/stdedos/matplotlib-stubs-hoel/actions/runs/6221692521/job/16884150094
And I'd like to:
- Track its progress - if it somehow "changes". I cannot do that right now, since
expected_failmeans "negative test" (instead of pytest'sxfail) - Revert "the useless todo". If it happens to match "exactly", then this will be triggered. But otherwise, the slightest deviation will continue masking this
This is (hopefully) illustrated by this example:
$ cat test_test.py
import pytest
EXPECTED = 2
XFAIL = 3
@pytest.mark.parametrize(
"result",
[
1,
pytest.param(EXPECTED, marks=pytest.mark.xfail(strict=True)),
pytest.param(XFAIL, marks=pytest.mark.xfail(strict=True)),
],
)
def test_increment(result):
assert result == EXPECTED
$ pytest test_test.py
========================== test session starts ==========================
platform linux -- Python 3.8.10, pytest-7.4.1, pluggy-1.3.0
configfile: pyproject.toml
plugins: mypy-plugins-3.0.0, dash-2.13.0, pudb-0.7.0, mypy-testing-0.1.1
collected 3 items
test_test.py FFx [100%]
=============================== FAILURES ================================
___________________________ test_increment[1] ___________________________
result = 1
@pytest.mark.parametrize(
"result",
[
1,
pytest.param(EXPECTED, marks=pytest.mark.xfail(strict=True)),
pytest.param(XFAIL, marks=pytest.mark.xfail(strict=True)),
],
)
def test_increment(result):
> assert result == EXPECTED
E assert 1 == 2
test_test.py:16: AssertionError
___________________________ test_increment[2] ___________________________
[XPASS(strict)]
======================== short test summary info ========================
FAILED test_test.py::test_increment[1] - assert 1 == 2
FAILED test_test.py::test_increment[2]
===================== 2 failed, 1 xfailed in 0.10s ======================
Activity