When using automation test scripts (whether Selenium or Appium), best practice includes the use of checkpoints to verify that the application is displaying the correct information. Manual testers perform this validation by visually examining the application display and verifying that they see all of the correct results.
Automation performs the validation by identifying the UI objects, based on XPath, UIAutomation, or similar tools. However, there are cases where this object analysis may not be possible. This is the case when:
- The objects are not well-defined, making it difficult to arrive at a conclusive validation.
- It is required to verify based on what is actually displayed ("what the user sees").
Perfecto provides a Visual Analysis tool based on analyzing a screenshot of the actual device screen (100% reflection of what the user actually sees).
Find a needle in the haystack
Using a well-known analogy, visual analysis can be viewed as finding a needle (the text or image) in a haystack (the screenshot). You can quickly create the code snippet for text or image analysis in the Manual testing view.
Text identification
Perfecto uses a third-party OCR tool to perform the analysis of the screenshot and to identify all occurrences of the text "needle" on the screen. If the text is identified, Perfecto will identify the UI element closest to the text and support actions, for example click or setText, for the element.
Image identification
Similarly, when the "needle" is an image, Perfecto's image visual analysis will analyze the screenshot in segments congruous with the size of the needle to try and identify the location of the image on the screen. To improve the success rate, we advise you to try and match the resolution of the needle to the device resolution or to use an image at the highest device resolution possible. To improve the speed of the analysis, we advise the use of the haystack size parameters to limit the area of the screen that is searched. This is very useful when the image is expected to appear in a specific area of the screen.
The needle image should be uploaded to the Perfecto Lab Repository for referencing from the Visual Analysis commands.
Use visual analysis in Selenium/Appium
To add Visual Analysis functions to your Selenium/Appium test scripts, use the executeScript() driver method as described here. Prior to using the image "needle", use the PerfectoLabUtils.uploadMedia method to copy the image to the repository.
Understand visual analysis parameters
The visual analysis commands/functions support a basic set of common parameters to control how the visual analysis is performed. Understanding some of these parameters will help you optimize the use of this powerful functionality.
Match mode parameter
The match parameter controls how the visual analysis compares the "needle" image to the screenshot ("haystack"). The following modes are supported:
-
Same Size – Needle and haystack have the same resolution, for example when the needle is taken from a screenshot on the same device.
-
Any Size – Haystack may be of any resolution. This increases analysis time.
-
Bounded Size – Provide limits on different resolutions, for example when using the same needle image on multiple devices. Limits increase analysis time.
Consider using the "Bounded size" value for the parameter unless the needle image was taken from the same device.
Screen dimension parameters
You can limit the area of the screenshot (haystack) examined by the visual analysis tool. This is especially useful if the image/text that you are looking for is known to appear in a particular area of the screen, for example the screen header or footer.
Visual analysis functions support the following screen dimension parameters:
|
Selenium/Appium |
Description |
|---|---|
|
screen.top |
Indicates the distance from the top of the screen to the top of the haystack rectangle |
|
screen.height |
Indicates the height of the haystack rectangle |
|
screen.left |
Indicates the distance from the left side of the screen to the left side of the haystack rectangle |
|
screen.width |
Indicates width of the haystack rectangle |
Essentially, provide the coordinates of the top-left corner of the haystack rectangle and the rectangle's dimensions. The parameters may be provided either in number of pixels or as a percentage of the screen resolution.
Scroll to find a needle
The Scroll and search (scrolling) parameter can be found in Visual Analysis commands such as Edit Set/Get and Button Click. It is used to search for a needle, image or text, by navigating across or down the device screen. This functionality exists only as a parameter and is activated when the needle is not found. See also General Visual Analysis Parameters.
The following parameters used to define a scroll are:
- Scroll and search (scrolling) - A Boolean value enabling a scroll until the needle is found (default false).
- Max scroll (maxscroll) - The maximum number of scroll actions to perform before returning (default 5).
- Next - How to perform the scroll action. Basic directions are SWIPE_UP, SWIPE_DOWN, SWIPE_RIGHT, and SWIPE_LEFT.
Scroll direction
The default scroll direction defined by the system is up. However, for iPhone devices, the default scroll direction is right. To change the scroll direction to down, select body as the Search area.
Before beginning your scroll, use a checkpoint to confirm that you are on the correct page to search for the needle.
Use wait with timeout
When enabling scrolling, the timeout must be zero. Scroll cannot be used together with a positive Timeout because this command dictates a different behavior when the needle is not found.
A Timeout is a defined time, in seconds, to wait for a needle to appear on the device screen. It can be used with a dynamic screen where the page loading time should be considered when searching for the needle. Using a scroll results in longer execution times.
