Exit with non-zero exit code when explicit test selection (`--filter`, `--group`, `--testsuite`) yields no tests · Issue #6276 · sebastianbergmann/phpunit · GitHub
Skip to content

Exit with non-zero exit code when explicit test selection (--filter, --group, --testsuite) yields no tests #6276

Closed
Closed
Exit with non-zero exit code when explicit test selection (--filter, --group, --testsuite) yields no tests#6276
Assignees
sebastianbergmann
Labels
feature/test-runnerCLI test runnertype/bugSomething is brokenversion/12Something affects PHPUnit 12version/13Something affects PHPUnit 13

Description

@staabm
staabm
opened on Jul 15, 2025
Q A
PHPUnit version a62124f
PHP version 8.3.23
Installation Method Composer / PHAR

How to reproduce

➜  phpunit git:(main) php phpunit --filter FilterWhichDoesNotMatchAnything          
PHPUnit 12.3-ga62124f38 by Sebastian Bergmann and contributors.

Runtime:       PHP 8.3.23
Configuration: /Users/m.staab/Documents/GitHub/phpunit/phpunit.xml

No tests executed!
➜  phpunit git:(main) echo $?            
0

➜  phpunit git:(main) php -v                                              
PHP 8.3.23 (cli) (built: Jul  1 2025 16:52:12) (NTS)
Copyright (c) The PHP Group
Zend Engine v4.3.23, Copyright (c) Zend Technologies

Expected behavior

I think in case we do not run any tests and had a --filter defined, PHPUnit should return a non-zero exit code

Activity

  1. staabm
    added
    type/bugSomething is broken
    on Jul 15, 2025

  2. staabm commented on Jul 15, 2025

    @staabm
    staabm
    on Jul 15, 2025
    ContributorAuthor

    I think the same should be true for --testsuite and --group based filters:

    ➜  phpunit git:(main) php phpunit --group test-abc
PHPUnit 12.3-ga62124f38 by Sebastian Bergmann and contributors.

Runtime:       PHP 8.3.23
Configuration: /Users/m.staab/Documents/GitHub/phpunit/phpunit.xml

No tests executed!
➜  phpunit git:(main) echo $?
0
➜  phpunit git:(main) php phpunit --testsuite test-abc
PHPUnit 12.3-ga62124f38 by Sebastian Bergmann and contributors.

Runtime:       PHP 8.3.23
Configuration: /Users/m.staab/Documents/GitHub/phpunit/phpunit.xml

No tests executed!
➜  phpunit git:(main) echo $?                         
0

  3. kubawerlos commented on Jul 15, 2025

    @kubawerlos
    kubawerlos
    on Jul 15, 2025
    Contributor

    What about running without any parameters and still getting "No tests executed!" (because of no tests)?

    What about a simple relation, "No tests executed!" -> "a non-zero exit code"?

  4. staabm commented on Jul 15, 2025

    @staabm
    staabm
    on Jul 15, 2025
    ContributorAuthor

    this might work, too.

    I tried to talk about my actual use-cases. but it might work for a more general case.

    not sure about possible BC impact though.

  5. joteejotee commented on Jul 16, 2025

    @joteejotee
    joteejotee
    on Jul 16, 2025

    I'd like to work on this issue if nobody else is working on it.

  6. staabm commented on Jul 16, 2025

    @staabm
    staabm
    on Jul 16, 2025
    ContributorAuthor

    We have no signal from Sebastian yet whether a fix would be accepted (he is on vacation atm).

    But if you want to work on it - go ahead.

  7. joteejotee commented on Jul 16, 2025

    @joteejotee
    joteejotee
    on Jul 16, 2025

    That makes sense—since we haven’t heard Sebastian’s opinion yet, I’ll hold off for now. Thanks for letting me know!

  8. stof commented on Jul 18, 2025

    @stof
    stof
    on Jul 18, 2025
    Contributor

    There is already a --fail-on-empty-test-suite config setting (the XML config file also supports it). Isn't it solving that need already ?

  9. staabm commented on Jul 18, 2025

    @staabm
    staabm
    on Jul 18, 2025
    ContributorAuthor

    Isn't it solving that need already ?

    thanks, wasn't aware of --fail-on-empty-test-suite.

    I think the behaviour of --fail-on-empty-test-suite should be the default for --filter (or maybe even in general).

  10. stof commented on Jul 18, 2025

    @stof
    stof
    on Jul 18, 2025
    Contributor

    @staabm just set failOnEmptyTestSuite="true" in your XML config file (or failOnAllIssues="true").

    Right now, none of the failOn* setting defaults to true, so this is consistent.

  11. staabm commented on Jul 25, 2025

    @staabm
    staabm
    on Jul 25, 2025
    ContributorAuthor

    thanks for the examples. that indeed proves that my initial problem is actually not the same as the feature --fail-on-empty-test-suite seems to provide.

    I think its a DX problem that PHPUnit reports a "success" result when its instructed to run with a certain filter but no tests match such filter. it would be useful for any tool (KI Agent, CI Job, batch file,..) invoking phpunit that no tests were run (by default, without an additional required config flag)

  12. sebastianbergmann
    changed the title [-]`--filter FilterWhichDoesNotMatchAnything` should exit with non-zero exit code[/-] [+]Exit with non-zero exit code when explicit test selection (--filter, --group, --testsuite) yields no tests[/+] on Apr 12, 2026
  13. sebastianbergmann
    self-assigned this
    on Apr 12, 2026
  14. sebastianbergmann
    added
    version/12Something affects PHPUnit 12
    version/13Something affects PHPUnit 13
    feature/test-runnerCLI test runner
    on Apr 12, 2026
  15. sebastianbergmann
    changed the title [-]Exit with non-zero exit code when explicit test selection (--filter, --group, --testsuite) yields no tests[/-] [+]Exit with non-zero exit code when explicit test selection (`--filter`, `--group`, `--testsuite`) yields no tests[/+] on Apr 12, 2026
  16. sebastianbergmann
    added a commit that references this issue on Apr 12, 2026

    Closes #6276

    bf6b5a9

  17. sebastianbergmann commented on Apr 12, 2026

    @sebastianbergmann
    sebastianbergmann
    on Apr 12, 2026
    Owner

    When --filter, --exclude-filter, --group, --exclude-group, --testsuite, or --exclude-testsuite is used and no tests match, PHPUnit now exits with exit code 1 instead of 0.

    The change is a one-line modification in ShellExitCodeCalculator: the default value of $failOnEmptyTestSuite is set to true when any of those options are active, instead of unconditionally false. This means the existing failOnEmptyTestSuite logic kicks in automatically when explicit test selection yields no results.

    --do-not-fail-on-empty-test-suite (and doNotFailOnEmptyTestSuite="true" in XML) still overrides this and restores exit code 0, so there is an escape hatch for anyone who needs the old behavior.

    Without any filter/group/testsuite selection, empty test suites continue to exit with 0, no change in behavior there.

    $ php phpunit --filter FilterWhichDoesNotMatchAnything
PHPUnit ...
No tests executed!
$ echo $?
1

$ php phpunit --group nonexistent-group
PHPUnit ...
No tests executed!
$ echo $?
1

$ php phpunit --testsuite nonexistent-suite
PHPUnit ...
No tests executed!
$ echo $?
1

$ php phpunit --filter FilterWhichDoesNotMatchAnything --do-not-fail-on-empty-test-suite
PHPUnit ...
No tests executed!
$ echo $?
0
  18. sebastianbergmann
    added a commit that references this issue on Apr 12, 2026

    Merge branch '12.5' into 13.1

    bfb92ce
  19. sebastianbergmann
    closed this as completedin bf6b5a9on Apr 12, 2026
  20. sebastianbergmann
    added 4 commits that reference this issue on Apr 12, 2026

    Merge branch '13.1'

    716ce3b

    Closes #6276

    ae1cf4f

    Merge branch '12.5' into 13.1

    f26c46e

    Merge branch '13.1'

    abefdcc
  21. Jean85
    added a commit that references this issue on Apr 13, 2026

    Fix tests that may trigger failures for empty test suites

    c2d84f9
  22. CybotTM
    added a commit that references this issue on Apr 25, 2026

    fix: tolerate empty Functional test suite (PHPUnit 12.5.18+)

    ecee1ee
  23. github-actions
    locked as resolved and limited conversation to collaborators on May 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

feature/test-runnerCLI test runnertype/bugSomething is brokenversion/12Something affects PHPUnit 12version/13Something affects PHPUnit 13

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions