Last updated: May 31, 2020 17:52
by Prasant Sutaria
At times, the number of tests executed as part of a test suite doesn't match with the job count in the CI Dashboard view. There can be several reasons for the mismatch, but the most common one is that for some reason the Driver is not initialized. The Smart Reporting SDK depends on the test driver. If, for some reason, the driver initialization fails, we may get the test instance missing from the Dashboard job.
Suggested Best Practise:
- Provide "scriptName" desired capability as part of driver initialization.
scriptName will provide default test name in case of driver initialization fails. This can help in reducing test cases with name "RemoteWebdriver".
Note: The test name provided by Digital Zoom API later can override the default test name provided by scriptName desired capability.
String scriptname = "General-Device-Test";
- Provide "report.jobName" and "report.jobNumber" desired capabilities as part of driver initialization.
report.jobName and report.jobNumber capabilities will help in providing default Job name and number the test belongs to. This can help in resolving the count mismatch issue.
Note: The report.jobName and report.jobNumber provided by Digital Zoom API later can override the default Job Name and Job Number provided by the mentioned desired capabilities.
- Verify that the testing framework is indeed creating logical tests for each test.
In some cases, frameworks are set up to skip or fail subsequent tests if a previous test within a test suite is failed. Ensure that Smart Reporting is notified about each test- either by ensuring a connection attempt to a device is sent (attempt to instantiate driver), or if an alive driver exists, by explicitly sending TestStart and TestStop failing the test if you do not want it to be executed. "Failure Reason" can be used to mark these as not requiring further analysis.
Failing to do above and just skipping tests depending on previous execution outcomes, will result in changing test counts and make analysis much harder.
Related Article: Define capabilities