Repository navigation
[BUG] Report results not stable #513
Description
Describe the bug
I have integrated my Reassure tests into a Bitbucket pipeline. My tests run on the pipeline, and I display a summary of my test report in the reports section on Bitbucket. However, the issue is that the tests do not consistently yield the same results when executed.
Reassurance base branch also works. Then it works in the branch where development is carried out and produces the results. This process is done every time a pipeline is triggered.
When I produce the result several times, there is sometimes a big difference between them.
To Reproduce
Steps to reproduce the behavior:
Go to the Bitbucket pipeline with Reassure tests.
Run the pipeline.
Again Run the pipeline
Compare the difference between two test results.
See the inconsistency in the test results.
Expected behavior
I expected the tests to produce consistent results every time they are run on the pipeline.
Screenshots
As shown in the images, the inconsistency in test results undermines my confidence in Reassure.
Activity
mdjastrzebski commentedon Jul 12, 2024
The first picture you attached seems to show reasonably stable results. Although I am not sure what exactly is running there, as this seems to be some custom report, not the one produced by Reassure by default.
The second picture seems to have some significant variation (~3k secs vs ~5k secs) in the first line. I can only guess where does it come from. My recommendation would be to make sure that your CI agent is not running any other jobs during the performance test runs as they will significantly impact your measurements. This is something that Reassure as a tool has no control over, as it's your CI setup.
yakupdurmus commentedon Jul 12, 2024
I understand, I guess the CI pipeline running with other jobs. In this case, running the code repeatedly doesn't yield the same results.
How can I ensure that only the Reassure performance test is being executed? Where and how are tests recommended to be run?