Page tree
Skip to end of metadata
Go to start of metadata

Last updated: Dec 12, 2018 14:28

Introduction

When the test automation script is run it may end in a failure. Based on the way the test failed, it may be useful to provide a failure reason that can visually identify the different test results when reviewing the tests in the DigitalZoom Report Library view. The DigitalZoom Reporting system provides functionality that supports generating this failure reason classification, either:

  • Manually by the test script, based on the stage in the script execution when the test is determined to have failed.
  • Automatically by DigitalZoom, that analyzes the entire test information and based on a heuristic classification generates the reason.

In either case, the failure reason is color-coded and displayed in the grid of the test reports, in the Failure Reason column to allow a quick overview of the different failed tests.

Note: Blocked Failure Reasons (see below) appear in yellow.


Blocked Failure Reasons

The DigitalZoom system will automatically identify instances when the test failed before the test was able to start. When these failures occur, the report will be marked with a blocked failure reason (that will appear in the Report Library Grid as the Failure Reason) and the Status column will show a Blocked status (). These blocked failure reasons are completely controlled (failure reason text and color) by the DigitalZoom system. Some examples of blocked failure reasons include:

  • Device in use

  • Device not connected

  • Device not found

Note: The complete list of blocked failure reasons do not appear in the Admin tab, see below.

  • Blocked reasons cannot be edited in any way by the customer.

Failure Reason Names

Perfecto pre-defines several failure reasons, including:

  • Element not found
  • Invalid XPath syntax
  • Multiple elements found
  • Popup handling

Each of the names has a color associated with it. The color is used by the DigitalZoom UI to more easily indicate the different test report failure reasons.

The user can define additional failure reason names. Pre-defined failure reasons can, currently, only be edited or deleted by Perfecto support.

Custom Failure Reasons

Adding a new Custom Failure Reason

To add a new custom failure reason name string:

  • Open the Admin tab of the reporting DigitalZoom interface.
  • Select the FAILURE REASONS tab (displays all of the existing failure reason names)
  • Click on the Add another failure reason link (at bottom of the list)
  • Enter the failure reason name string (maximum 25 characters) in the text box
    • Select a color for the failure reason
    • Click the check-mark to add the name, or the X to cancel the addition.

Note: System supports a maximum of 8 additional (beyond those supplied by default) failure reason names.

Custom Failure Reason IDs

Each custom failure reason string is assigned a unique ID. This ID is used by the test scripts to indicate what caused a test to fail (see below).

To see the ID assigned to a particular failure reason -

  • Open the list of defined failure reasons, by opening the FAILURE REASONS tab of the Admin area.
  • Hover over the failure reason string - a tooltip displays the assigned ID.

To copy the ID either directly into a test script, or into a text file used by the test scripts to obtain the different failure reason IDs -

  • Hover over the row of the custom failure reason - a copy to clipboard icon () is displayed at right end of row.
  • Hover over the icon to display a tooltip that displays the failure reason ID.
  • Click the icon to copy the ID to the clipboard.
  • Paste the ID into the destination text (or code) file.

Using Custom Failure Reasons in Test Scripts

Both the DigitalZoom SDK and the External Test Information SDK support supplying the failure reason, for a test script or report, using the TestResultFactory's createFailure() method.

The method supports three parameters:

  • A failure message that will be added to the test report.
  • The actual error notification - throwable entity.
  • A failure reason ID String. The string must match one of the defined failure reason IDs (see above) . If the provided string does not match any of the defined failure reasons, it will be ignored.

The following example shows how to create the failure TestResult with a failure reason (one of the basic failure reasons supplied by Perfecto):

TestResult testResult = TestResultFactory.createFailure("Test stop failure", e, "bEC3xAdMTz");  //Paste the failure reason ID after copying from the Admin tab
reportiumClient.testStop(testResult);

See a full example test code here.