Page tree
Skip to end of metadata
Go to start of metadata

Last updated: Oct 23, 2018 09:24

Background

Automation at scale that implements Responsive Web Design or Parallel execution, requires advanced tools for fast triaging.

The need is increasingly growing as these testing methodologies involve execution across multiple digital platforms, in different screen orientations, and under different environment conditions. Visual validation and comparison of test execution results, is a natural solution for an effective triaging process and to easily highlight the areas of risk.

With the Cross Report View functionality, you can make sure that the new code tested is rendering properly across all digital platforms.

Comparing Test Results Visually with Cross Report Display

Selecting tests for Custom Cross Report

To compare the test results by displaying them "Side-by-Side", start from the Report Library View of the DigitalZoom Reporting interface. Using the check-boxes, to the left of each test name, select the tests that you want to compare. Or use the test-set indicator (see below) to select the whole test-set.

Note: Tests whose report artifacts are older than the test retention period, and tests that use more than one platform cannot be selected for Cross Report display.

When you select the first test, by check-box, for the cross-report:

  • A new header line is displayed - that includes:
    • Number of test items selected.
    • A button to display the Custom Cross Report View.
    • A button to clear the selection of the tests.

To display the Custom Cross Report View - click the button labelled Compare now.

Using the Perfecto Report Cross Report View

Test-set Definition

Test-set: DigitalZoom will automatically define groups of tests that have run in the last 7 days and have the same defined characteristics (including job, project, branch information) but may differ in the capabilities, and especially device selection capabilities. Members of a test-set that share the same set of capabilities, will be represented by the latest test-run.

When DigitalZoom included a test-run into a test-set - the following icon () is displayed in the Perfecto Report indicator column (see the screenshot above) of the Report Library View. The icon is displayed for all tests in the test-set.

Clicking on the icon displays the test-specific Cross Report View for the test-set (see below).

Cross Report View

Custom Cross Report View - Common Steps

DigitalZoom analyzes the test reports selected for the Cross Report view and identifies the set of steps used for the tests. When common steps are identified, the Cross Report View displays -

  • Top Header Line - statistical summary of the tests selected for the Cross Report
  • Second Header Line - short summary of each test selected, in clickable widgets, that includes:
    • Device type and OS used for the test
    • Final test status
    • Date test run
  • Left Panel - common steps of the different test reports.
  • Right Panel - set of screenshots from each of the selected test reports. The screenshot corresponds to the state of the device/test for the step selected in the left panel.

Usage Notes:

  • Clicking on the Statistical Overview fields (in Top Header line) will filter the displayed tests to focus on all tests with the selected final status (Pass, Fail, or Unknown).
  • The screenshot panels are only displayed if one of the steps is selected. When first displayed the Cross-Report display does not have any step selected.
  • The Report Status Summary widgets and the screenshot panels are coordinated - hovering over any of them highlights the corresponding pair.
  • Clicking on either the summary widget or the screenshot panel opens the Single Test Report selected.
  • if one of the tests did not execute one of the steps - the screenshot panel for the step will not display a screenshot and will note that the step was not executed.

Custom Cross Report View -  No Common Steps

When no common steps (between the selected tests) were identified, the Custom Cross Report View displays the final screenshot of the selected test reports, in a preview frame per test. The tests are displayed side-by-side to allow the user to easily compare the test conditions and results.


Note: If test has "Passed" state, the last command executed is displayed (screenshot and step name). If the test has "Failed" status, display the last command (screenshot and step name) that reported failure.

Test-specific Cross Report

In most cases a Test-specific Cross Report will present the Common Steps view (see above). In rare cases, when tests with the same name run slightly different scenarios, the test-specific Cross Report will display without the common steps.

In either case, the Test-specific Cross Report includes the same information as the Custom Cross Report, with the following notes:

  • The window's header includes the Test Name for the test-set.
  • The status line and the preview frames are displayed for all tests included in the test-set.

Where do you go from here

Click in the top part of the preview frame to go to the Single Test Report for the specific test run.

There is one change to the Single Test Report when coming from the Custom Cross Report view:

  • Addition of "breadcrumb" button to return to the Cross Report view next to the button to return to the Report Library.
  • When DigitalZoom identified a test-set (as described above), the return button, in the breadcrumbs, will return to the test-specific Cross Report.