The left panel of the STR view shows a list of the logical steps that comprise the test (as named in the testStep() method).
Reports for native automation executions that activate nested scripts include the steps of both the main script and the nested script. The commands of the nested scripts are identified by a special symbol ("
You can click a command to display detailed information about the command execution, including:
The right panel of the STR view presents visual artifacts, such as screenshots or videos from the test run.
When you view the video, the timeline includes indicators that highlight the times at which the logical steps occurred. Moving the pointer over any of these points displays a tooltip that identifies the corresponding logical step.
When the test script displayed in the STR generated an error or failure message, the message is displayed at the top of the visual artifact area. At first, only the header line of the error message is displayed on a red background. To see the full message, together with a stack dump (if relevant), click the down arrow below the error message:
The Single Test Report header consists of two lines that include the following information:
Perfecto can be configured to collapse scheduled retries into a single test report. This feature is turned off by default. To turn it on in your cloud instance, contact Perfecto Support.
For a test to be considered a retry, it must share the same parameters and CI job name and number or be part of the same execution. Perfecto does not list a test that is considered a retry in the table and does not take it into account when calculating statistics. Only the last test in a retry series makes it into the statistics.
If the current test is part of a scheduled retry, this is indicated by two arrows surrounding the double-ringed current-run icon (). In this case, the first header line includes the retry navigation bar (). You can use this bar to scroll through the individual retries.
The following video illustrates how the scheduled retry feature works.
When the test script or Smart Reporting heuristics have identified a failure reason, it appears in the second line next to the test status (only if the status is Failed). The failure reason is one of those configured for the CQ Lab by the administrator.
If no failure reason was identified for a failed test, the Add failure reason button is displayed instead.
To add a failure reason:
To update a failure reason:
Use the Report Details button in the upper right corner of the STR view to display the Report Details form. This form shows information related to the selected test, as follows:
Note: If provided, the EXECUTION tab information for a test run that failed during Driver creation, prior to ReportiumClient instantiation, is taken from the capabilities settings (for Selenium/Appium scripts).
Sometimes, a test run does not complete as expected and results in a status of Failed. In such a case, it is often easier to understand what went wrong or what needs to be fixed in the test script if you can view the source code. This functionality is dependent upon the tester supplying the information, as described in Access source code. Perfecto displays source code in a new browser tab.
Links to source code are configurable via custom fields set by the test run.
If the test run sets both custom fields, both fields display (Open source file and Open commit field).
To open the source code from the STR:
You can download a PDF version of the test report to your local workstation through the download menu, accessible from the download button at the top right.
Test reports include the basic information on the test execution displayed in the STR. In addition, the test may attach other log files as artifacts, such as of the device activity, network activity, or other vitals information.
There are times when a test fails because the application under test crashes. This is actually a case where the test succeeded in uncovering an application fault. When this occurs, the mobile operating system generates a crash log that includes information that the application developer can then use to identify the cause of the crash.
The Perfecto system identifies these situations, retrieves the crash log from the device, and notifies the Smart Reporting analytics of the status. Smart Reporting identifies the failure reason for this type of test as Application crashed. It add the crash log as an artifact to the test report.
This is supported on all iOS versions, Samsung devices running Android 7.0 and later, and other Android devices running Android 6.0 and later.
To retrieve the crash report for the test:
When a Perfecto Native Automation test script allocates multiple devices to run the test, the reporting system gathers artifacts (screenshots, video) from all of the devices involved. At the completion of the test execution, the report for the test run generates a single report that includes the artifacts from all devices.
Note: Displaying a single report for multiple devices is available only for Perfecto Native Automation.
The Report Library view lists test runs that activated multiple devices with the following indications that multiple devices were involved:
The STR of a test that activated multiple devices includes the following indications that multiple devices where involved:
Screenshots are available for all devices involved.