Page tree
Skip to end of metadata
Go to start of metadata

Last updated: Oct 14, 2019 13:00

In this section:

When running your Instrumentation tests with the Perfecto Gradle Plugin, you can add configuration information for the test execution at three levels:

  • Configuration file - a JSON formatted file listing parameter assignments - (see below).
  • In the build.gradle file - in the perfectoGradleSettings clause, include Authentication parameters and indicate the location of the configuration file.
  • On the command-line - include any of the parameters in format: -P<paramName>="<paramvalue>".

Configuration Parameters

The configuration parameters are grouped into the following categories:

Perfecto Lab Authentication

ParameterPossible ValuesMeaning
cloudURL
URL of the Perfecto Lab to connect to. For example, mobilecloud.perfectomobile.com
securityTokenSecurity TokenTester's personal security token for the Perfecto Lab.

These parameters may be set:

  • In the configuration file, in the following format:

    "cloudURL": "mobilecloud.perfectomobile.com"
    "securityToken": "AAABNg0ODAoPeNqtkT1PwzAQhnf/CkssM...JxQ3HEI8NsX02ff"
  • In the build.gradle file, in the following format:

    perfectoGradleSettings { 
        cloudURL "mobilecloud.perfectomobile.com" 
        securityToken "AAABNg0ODAoPeNqtkT1PwzAQhnf/CkssM...JxQ3HEI8NsX02ff" 
    } 
  • In the command-line in the following format:

    > gradle perfecto-xctest -PcloudURL="mobilecloud.perfectomobile.com" -PsecurityToken="AAABNg0ODAoPeNqtkT1PwzAQhnf/CkssM...JxQ3HEI8NsX02ff"

Device Selection

Selecting specific devices

You can select to either use the following parameters to specify a list of devices -

Obtain the values for the following parameters based on the information in the Perfecto Lab Launch Mobile window as listed for the specific devices to select.

ParameterPossible ValuesMeaning
deviceName
The device ID
description
The device description as defined in the Perfecto Lab.
location
The physical location of the device.
manufacturer
For mobile devices manufacturer of the model. For example: Apple.
model
Device model (i.e. iPhone-7)
network
Device network (i.e. AT&T, Verizon)
platformNameiOS | AndroidDevice operating system
platformVersion
Operating system version - for example 11.0.3
resolution
Screen resolution of the device
Note: If no device selection parameters are set, the plugin will select a random device, from the devices connected to the Perfecto Lab.

These parameters may be set in the configuration file, in the following format:

  • Single device selection:

    "devices":
    [{
        "platformName" : "Android"
        "platformVersion": "^[678].*"
        "manufacturer" : "Samsung"
    }]
  • Multiple device selection (in the following example, four devices are selected):

    "devices" : [
    	{"deviceName" : "FA6BR0304878"},
    	{
    		"platformVersion": "11.*",
    		"manufacturer" : "Apple"
    	},
    	{
    		"model": "iPad 4",
    		"location": "US-MA-BOS",
    		"network": "AT&T, Verizon"
    	},
    	{}
    ]

    Note: The fourth device in this example - "{}" is chosen randomly from all available iOS devices.

Specifying a Number of Random Devices

You can use the following parameter, in place of the devices clause, to indicate that the specified number of iOS devices should be selected and the Espresso tests run on the devices in parallel.

ParameterPossible ValuesMeaning
numOfDevicesnumber in range [1,100]Number of devices to select. If numOfDevices is outside the range, the value will default to select a single device.

This parameter may be set in either the configuration file, the perfectoGradleSettings clause of the build.gradle file, in the following format:

"numOfDevices" : 45

or on the command line, in the following format:

> gradle perfecto-android-inst ... -PnumOfDevices=33 ...

Selection Retry Parameters

If the plugin was unsuccessful in allocating the device, you can use the following parameters to configure a retry mechanism built into the plugin:

ParameterPossible ValuesMeaning
acquireDeviceRetryNumber default = 0Number of retries that plugin should attempt to allocate the device.
acquireDeviceRetryInterval default = 30Number of seconds to wait between each retry attempt

These parameters may be set in either the configuration file, the perfectoGradleSettings clause of the build.gradle file, or on the command line, in the following format:

Configuration File format
"acquireDeviceRetryNumber" : 5
"acquireDeviceRetryInterval" : 45
build.gradle file
perfectoGradleSettings { 
	...
    acquireDeviceRetryNumber  5
	acquireDeviceRetryInterval  45 
} 
Command Line format
gradle perfecto ... -PacquireDeviceRetryNumber=5 -PacquireDeviceRetryInterval=45

Network Virtualization

The Gradle plugin supports configuration of a virtual network environment to execute the XCUITest/XCTest tests.

Use the networkVirtualization clause in your configuration file or build.gradle file to define the parameters to define the characteristics of the emulated network. The virtual network can be configured using the same parameters supported by the Perfecto Network virtualization start command. The following is the list of parameters supported:

Sub-ParameterPossible ValuesMeaning
latencyIn the range 0-8000 ms.Latency applied on packets in the Network.
packetLossIn the range 0-100%Network packet loss.
A reasonable packet loss value should not exceed 5%.
bandwidthInIn range 3-100,000 Kbps
or unlimited.
Limitation on the allowed download network bandwidth into the device.
bandwidthOutIn range 3-100,000 Kbps
or unlimited.
Limitation on the allowed upload network bandwidth from the device.
packetCorruptionIn the range 0-100%Network packet corruption.
A reasonable packet loss value should not exceed 5%
packetReorderingIn the range 0-100%The percentage of network packets sent immediately without any delay.
Used alongside the latency parameter. Moreover, the packets sent
immediately will arrive earlier than the packets that were delayed by the
defined latency value, essentially creating a packet reordering.

A reasonable packet reordering value should not exceed 10%.
packetDuplicationIn the range 0-100%Network packets duplicated.
A reasonable packet duplication value should not exceed 3%.
delayJitterIn the range 0-8000 ms.Random latency variation; the actual latency between latency +- jitter.
Used alongside the Latency parameter.

For example, if the latency is defined to be 100 ms and
the jitter is 10 ms, this causes the added delay to be 100ms - 10ms.
correlationIn the range 0-100%Network packet value correlation, affecting the Latency, Corruption,
Reordering and Duplication.

For Latency, Corruption, Reordering and Duplication, the current packet
n value will be correlated by this % to previous packet n-1 value.
In other words, the current packet value is correlated to the previous
packet value.

For example, if the correlation is defined to be 25%, the packet loss is 5%,
and the previous packet was lost, then the updated packet loss value
(for the current packet) would be 75% * 5% + 25%, which equals 28.75%.
However, if the previous packet was not lost, then the packet loss value
would be 75% * 5%, which equals 3.75%.
blockedDestinationsList of StringsNetwork packet block, to specific destinations, defined by domain name,
IP address, and IP range destinations in IP Prefix (Slash) notation.
blockedPortsList of StringsNetwork packet block, to specific ports.
For example, to block http, define port number 80.
To unblock, prefix the value with a '-'. For example, -80.
generateHarFiletrue | false

Indicates if a HTTP Archive (HAR) file should be generated for
each test to analyze the traffic of the virtual network. See Note below.

profile
Suggested network virtualization profiles.
See the Network Conditions for supported values 

Note: To use the generateHarFile parameter, first install the certificate and configure the device using Perfecto Automation. This feature is currently in limited release, contact support

These parameters may be set:

  • In the configuration file, in the following format:

    "networkVirtualization": {
        "latency": 89,
        "packetLoss": 7,
        "bandwidthIn": 5,
        "bandwidthOut": 40
    }
  • In the perfectoGradleSettings clause of the build.gradle file:


    perfectoGradleSettings { 
        ...
        networkVirtualization {
            packetLoss 5
            profile    4g_lte_good
        }
        ...
    }

Location mock

The Gradle plugin supports providing a location for the device when executing XCUITest/XCTest and Espresso tests.

Use the locationMock clause in your configuration file or build.gradle file to define the parameters to define the characteristics of the emulated location. The following is the list of parameters supported:

Sub-ParameterPossible ValuesMeaning
addressValid addressAddress to be set on the device. The address will be automatically translated to latitude, longitude coordinates by the system.
latitudemax/min +90 to -90Latitude of the location
longitudemax/min +180 to -180
Longitude of the location

Note: Only one of the options should be provided. Use either the "address" clause or the "latitude", "longitude" clauses, but not both!


Examples:

  • For setting the location of the device to Amal St 13, Rosh Haayin, Israel, use the following configuration:

    "locationMock" : {"address" : "Amal St 13, Rosh Haayin, Israel"}
  • For setting the location of the device to the explicit coordinates 32.106710, 34.969384, use the following configuration

    "locationMock" : {"latitude" : "32.106710", "longitude" : "34.969384"}

Obtaining Google coordinated for an address

Use Google Maps to easily get the coordinates of a place.


Troubleshooting

If the Set device location command does not work:

  • Open, close, and reopen the application in which the location is set.

Android Troubleshooting

Verify the following settings in the device:

  1. Allow mock locations:
    Go to Settings > Developer options > Allow mock locations
    Android 6: Enable mock locations
  2. High accuracy:
    Go to: Setting > Location > Mode > High accuracy

Device/Application Vitals

The plugin supports collecting data regarding the underlying performance of either the application or the device. The vitals information may be used to identify extreme behavior of the application or device. Use the following parameters to indicate the vitals data to collect for the test run:

ParameterPossible ValuesMeaning
vitalssee list in Vitals start command description
The Vitals to collect.
intervalRecommended: between 10 and 30.The data collection frequency, in seconds. Default: 1 sec

These parameters may be set:

  • In the configuration file, in the following format:

    "vitalsMonitoring": {
        "vitals": ["outputs.monitors.memory.used", "outputs.monitors.cpu.total"],
        "interval": 12
    }
  • In the perfectoGradleSettings clause of the build.gradle file:

    perfectoGradleSettings { 
    	...
    	vitalsMonitoring {
    		vitals   "all"
    		interval 30
    	}
    	...
    }

Application Parameters

ParameterPossible ValuesMeaningNotes
instrumentationArgs

Custom arguments to pass to the Instrumentation Runner.

format:

  • Configuration file:
    "instrumentationArgs" : ["value","value",...]
  • Command line:
    -PinstrumentationArgs="value;value;..."

failBuildOnFailure true | falseFail the Gradle build if any test fails or there is a device error. Default is false.

tunnelId

Identifier of the tunnel created by Perfecto Connect. Value should be taken from the response of the perfectoConnect command.

Note: Perfecto Connect is a limited access feature. Contact Perfecto support to add to your Perfecto Lab.


debug
Run the task in debug mode to output more verbose messages.
appPath

Path to the ipa or app file for the application and unit tests.
If the parameter links to an .app file (XCode output), the plugin will first convert this to an .ipa file before installing on the device. For details, see Run XCUITests in the Perfecto lab.


testAppPath
Path to the UI test application runner ipa or app file.
If the parameter links to an .app file (XCode output), the plugin will first convert this to an .ipa file before installing on the device. For details, see Run XCUITests in the Perfecto lab.

shardtrue | false

Indicates that test runs should use the Sharding tests capability of the Android JUnitRunner. The number of shards to create is equal to the number of devices selected by the configuration.

Default: false


Note: The instrumentationArgs indicates that the test methods use an external framework and the associated screenshot tool. For example:

  • When using the KIF framework - use "instrumentationArgs" : ["KIF"]
  • When using the EarlGrey tool - use "instrumentationArgs" : ["EARLGREY"]

Format of appPath/testAppPath values

The path parameters indicate the location of the ipa files of the application and the test-application. These may be located in either the local workstation storage or in the Perfecto Lab Repository. To differentiate between these locations use the following format to indicate a location in the Repository:

repository://[PUBLIC | PRIVATE]:/<path to item>

Repository item must have .ipa type indicator.

The "repository://" prefix is mandatory for Repository locations.

Selective testing parameters

In addition, there are filter parameters that limit the test scenarios executed during the run:

ParameterPossible ValuesMeaningNotes
testClassNames

Array of class names that you wish to run. If not specified, all the classes will run. 

format:

  • Configuration file -
    "testClassNames":["Class","Class",...]
  • Command line -
    -PtestClassNames="Class;Class;..."

 testMethodNames

Array of method names that you wish to run. If not specified, all methods will run. 

format:

  • Configuration file -
    "testMethodNames":["Class#Method","Class#Method",...]
  • Command line -
    -PtestMethodNames="Class#Method;Class#Method;..."

testTimeoutnumber of milliseconds

An amount of time allocated to each test case.

If a test case exceeds the timeout, it will be reported as "failed". The full test set will continue with the next test case.


When activating the plugin, with runUITests and runUnitTests set true, supply both the appPath and testAppPath parameters.

These parameters may be set:

  • In the configuration file, in the following format:

    "testAppPath": "Users/usern/AndroidStudioProjects/Playground/app/build/outputs/apk/app-debug-AndroidTest.apk"
    "apkPath": "Users/usern/AndroidStudioProjects/Playground/app/build/outputs/apk/app-debug.apk"
    "testSize": "small"
    "testMethodNames": ["com.perfectomobile.com.DemoClass#demoMethod","com.perfectomobile.com.DemoClass#demoMethod2"]
    "androidOrchestrator" : tru
  • In the perfectoGradleSettings clause of the build.gradle file:

    perfectoGradleSettings { 
    	...
        testAppPath "Users/usern/AppProjects/Playground/app/build/outputs/app-Runner.app"
    	appPath "Users/usern/AppProjects/Playground/app/build/outputs/app.ipa" 
    } 
  • In the command line, in the following format:

    gradle perfecto-xctest ... -PappPath="Users/usern/AppProjects/Playground/app/build/outputs/app-target.ipa" -PrunUnitTests="false" 

    Note: tunnelId (for Espresso tests) cannot be supplied as a command line parameter.

KIF/EarlGrey Tests

When running test methods from External Frameworks, for example Earl Grey or KIF, based on the XCTest framework, the plugin considers these to be unit tests, even though they may in fact perform UI automation and testing. Therefore:

  • Configure the runUnitTests parameter value to true.
  • If not including any XCUITest test methods, set runUITests parameter value to false.
  • Configure the instrumentationArgs parameter to identify the framework (see note above).
  • Configure the framework to save the screenshots as described here.

Pre-Execution and Post-Execution Parameters

The following parameters control whether the application and test applications are uninstalled from the devices, either prior to installing the latest version or at the end of the tests:

ParameterSub-parameterPossible ValuesMeaningNotes
installationDetails

The parameters in this Clause affect the installation phase of the test run.
preCleanUptrue | falseIf the parameter is set (true) then the ipa files of the application and test-application are uninstalled before installing the new version (supplied by above parameters)
grantAlltrue | falseIf the parameter is set (true) then the application is granted all Android permissions listed in the application manifest
Note: Applicable to Android 6.0 or later.
Only relevant for Espresso application tests
preExecution--

The parameters in this Clause affect the device used for the test run.
reboottrue | falseIf the parameter is set (true) then the device will perform a reboot operation prior to installing the ipa files. Take into account that this is a time-consuming action that may delay the execution of the tests.
screenOrientationlandscape | portraitConfigures how to orient the device when executing the tests. This affects the application display, test execution, dashboard display, and report video.
postExecution-

The parameters in this Clause affect the post-execution phase of the test run.
uninstalltrue | falseIf the parameter is set (true) then the ipa files of the application and test-runner are uninstalled after the test has completed execution.

These parameters may be configured:

  • In the configuration file, in the following format:

    "installationDetails" : {
    						"preCleanUp" : true
    						"resign" : true
    						}
    "postExecution" : {"uninstall" : false}
    "preExecution" : {"screenOrientation" : "portrait"}
  • In the perfectoGradleSettings clause of the module's build.gradle file:

    perfectoGradleSettings { 
    	...
        postExecution {
    		uninstall false
    	}
    	installationDetails {
    		preCleanUp true
    	}
     } 

Screenshots

Screenshots are an important tool for analyzing the test actions, the following parameters affect XCUITest test executions:

ParameterPossible ValuesMeaningNotes
takeScreenshotOnTestEndtrue | falseSave screenshot of device at end of the test executions, regardless of the test result status.
takeScreenshotOnTestFailuretrue | falseSave screenshot of device at end of the test executions, if the test result status is a failure status.

These parameters may be set in the configuration file, build.gradle file, or the command-line.

Note: (iOS applications) These configuration parameters are applicable only to XCUITest test methods. For XCTest unit tests, these may be snapped by the test code, using third-party tools and retrieved by the Gradle plugin for the execution report.

Reporting Parameters

The following parameters are used by Smart Reporting to classify the execution report and make it easy to identify the reports for the execution.

ParameterPossible ValuesMeaning
tags
Set of tags to associate with the execution
jobName

CI identification of the build, used for classification of the report in the CI Dashboard.

jobNumber 
CI Job Number of the build
projectName
Name of the project - for classification
projectVersion
Version number assigned to the project for this build
branch
Branch name. like additional tag.

These parameters may be set:

  • In the configuration file, in the following format:

    "tags" : ["plugin", "unit-test", "demo"]
    "projectName": "playground"
    "projectVersion": "1.5"
    "jobName": "newFeature"
    "jobNumber": "45"
  • In the perfectoGradleSettings clause of the build.gradle file:

    perfectoGradleSettings { 
    	...
        projectName "playground"
    	projectVersion "1.5"
    } 
  • In the command line, in the following format:

    gradle perfecto-xctest ... -Ptags="plugin;uiTest;demo" -PjobName="newFeature"

Exporting the DigitalZoom Report

The Perfecto toolset generates a test execution report that can be exported for analysis. It is recommended to:

  1. Use these configuration parameters to define tags, jobName, and jobNumber for the test execution.
  2. Export the report data using these information items with the DigitalZoom Public API .

Configuration File Location

The plugin execution reads the configuration file whose location is indicated by the configFileLocation parameter that may be supplied:

  • In the perfectoGradleSettings clause of the build.gradle file.

    perfectoGradleSettings { 
    	...
        configFileLocation "C:\temp\XcuiTest\ConfigFile.json" 
    } 
  • Or in the command line

    gradle perfecto-xctest ... -PconfigFileLocation="C:\temp\XcuiTest\ConfigFile.json"

If the parameter is not supplied -

  • If running in Standalone mode or the file is not found at the default location, then tests will run on a randomly selected available device.
  • If running in Android Studio project - default location is <project dir>/src/main/assets/perfecto/instrumentedTest/configFile.json
  • The application and runner identification parameters (appPath and testAppPath) must be supplied in either the build.gradle file or on the command line.



Also in this section: