[IMPORTANT NOTICE]

------------------

Failed tests usualy indicate a problem with your local system setup

and not within PHP itself (at least for official PHP release versions).

You may decide to automaticaly submit a test summary to our QA workflow

at the end of a test run.

Please do *not* submit a failed test as a bug or ask for help on why

it failed on your system without providing substantial backup information

on *why* the test failed on your special setup. Thank you :-)

[Testing Basics]

----------------

The easiest way to test your PHP build is to run "make test" from the

command line after successfully compiling. This will run the complete

tests for all enabled functionalities and extensions using the PHP

CLI binary.

To execute test scripts, you must build PHP with some SAPI, then you

type "make test" to execute all or some test scripts saved under

"tests" directory under source root directory.

Usage:

make test

"make test" basically executes "run-tests.php" script

under the source root (parallel builds will not work). Therefore you

can execute the script as follows:

TEST_PHP_EXECUTABLE=sapi/cli/php \

sapi/cli/php [-c /path/to/php.ini] run-tests.php [ext/foo/tests/GLOB]

[Which "php" executable "make test" look for]

---------------------------------------------

If you are running the run-tests.php script from the command line (as above)

you must set the TEST_PHP_EXECUTABLE environment variable to explicitly

select the PHP executable that is to be tested, that is, used to run the test scripts.

If you run the tests using make test, the PHP CLI and CGI executables are

automatically set for you. "make test" executes "run-tests.php" script with the CLI binary. Some

test scripts such as session must be executed by CGI SAPI. Therefore,

you must build PHP with CGI SAPI to perform all tests.

NOTE: PHP binary executing "run-tests.php" and php binary used for

executing test scripts may differ. If you use different PHP binary for

executing "run-tests.php" script, you may get errors.

[Which php.ini is used]

-----------------------

"make test" uses the same php.ini file as it would once installed.

The tests have been written to be independent of that php.ini file,

so if you find a test that is affected by a setting, please report

this, so we can address the issue.

[Which test scripts are executed]

---------------------------------

"run-tests.php" ("make test"), without any arguments executes all

test scripts by extracting all directories named "tests"

from the source root and any subdirectories below. If there are files,

which have a "phpt" extension, "run-tests.php" looks at the sections

in these files, determines whether it should run it, by evaluating

the 'SKIP' section. If the test is eligible for execution, the 'FILE'

section is extracted into a ".php" file (with the same name besides

the extension) and gets executed.

When an argument is given or TESTS environment variable is set, the

GLOB is expanded by the shell and any file with extension "*.phpt" is

regarded as a test file.

Tester can easily execute tests selectively with as follows.

Examples:

./sapi/cli/php run-tests.php ext/mbstring/*

./sapi/cli/php run-tests.php ext/mbstring/020.phpt

[Test results]

--------------

Test results are printed to standard output. If there is a failed test,

the "run-tests.php" script saves the result, the expected result and the

code executed to the test script directory. For example, if

ext/myext/tests/myext.phpt fails to pass, the following files are created:

ext/myext/tests/myext.php - actual test file executed

ext/myext/tests/myext.log - log of test execution (L)

ext/myext/tests/myext.exp - expected output (E)

ext/myext/tests/myext.out - output from test script (O)

ext/myext/tests/myext.diff - diff of .out and .exp (D)

Failed tests are always bugs. Either the test is bugged or not considering

factors applying to the tester's environment, or there is a bug in PHP.

If this is a known bug, we strive to provide bug numbers, in either the

test name or the file name. You can check the status of such a bug, by

going to: http://bugs.php.net/12345 where 12345 is the bug number.

For clarity and automated processing, bug numbers are prefixed by a hash

sign '#' in test names and/or test cases are named bug12345.phpt.

NOTE: The files generated by tests can be selected by setting the

environment variable TEST_PHP_LOG_FORMAT. For each file you want to be

generated use the character in brackets as shown above (default is LEOD).

The php file will be generated always.

NOTE: You can set environment variable TEST_PHP_DETAILED to enable

detailed test information.

[Automated testing]

If you like to keep up to speed, with latest developments and quality

assurance, setting the environment variable NO_INTERACTION to 1, will not

prompt the tester for any user input.

Normally, the exit status of "make test" is zero, regardless of the results

of independent tests. Set the environment variable REPORT_EXIT_STATUS to 1,

and "make test" will set the exit status ("$?") to non-zero, when an

individual test has failed.

Example script to be run by cron(1):

========== qa-test.sh =============

#!/bin/sh

CO_DIR=$HOME/cvs/php5

MYMAIL=qa-test@domain.com

TMPDIR=/var/tmp

TODAY=`date +"%Y%m%d"`

# Make sure compilation enviroment is correct

CONFIGURE_OPTS='--disable-all --enable-cli --with-pcre'

export MAKE=gmake

export CC=gcc

# Set test environment

export NO_INTERACTION=1

export REPORT_EXIT_STATUS=1

cd $CO_DIR

cvs update . >>$TMPDIR/phpqatest.$TODAY

./cvsclean ; ./buildconf ; ./configure $CONFIGURE_OPTS ; $MAKE

$MAKE test >>$TMPDIR/phpqatest.$TODAY 2>&1

if test $? -gt 0

then

cat $TMPDIR/phpqatest.$TODAY | mail -s"PHP-QA Test Failed for $TODAY" $MYMAIL

fi

========== end of qa-test.sh =============

NOTE: the exit status of run-tests.php will be 1 when

REPORT_EXIT_STATUS is set. The result of "make test" may be higher

than that. At present, gmake 3.79.1 returns 2, so it is

advised to test for non-zero, rather then a specific value.

[Creating new test files]

-------------------------

Writing test file is very easy if you are used to PHP.

See the HOWTO at http://qa.php.net/write-test.php

[How to help us]

----------------

If you find bug in PHP, you can submit bug report AND test script

for us. You don't have to write complete script, just give us test

script with following format. Please test the script and make sure

you write the correct ACTUAL OUTPUT and EXPECTED OUTPUT before you

submit.

<?php

/*

Bug #12345

substr() bug. Do not return expected string.

ACTUAL OUTPUT

XYXA

EXPECTED OUTPUT

ABCD

*/

$str = "XYZABCD";

echo substr($str,3,7);

?>