Page tree
Skip to end of metadata
Go to start of metadata

Last updated: Dec 31, 2018 16:54

Activating the Plugin Task

To activate the plugin - Open a command-line window in the project's folder and run the perfecto-android-inst task in gradle:

gradle perfecto-android-inst <configuration parameters>

This will

  • Select the devices as specified in the Device selection parameters of the configuration file [or a random device if no specification provided],
  • Install the application and test apk files onto the device
  • Run the test methods (based on the configuration parameters)
  • Send output to the gradle console
  • Generate an execution report that can be viewed in the DigitalZoom Reporting interface.

The command line parameters can set any of the configuration parameters except for device selection parameters. For more information on the configuration parameters read this page.

Running over a proxy connection

If the Perfecto Lab is connected via a proxy, you will need to either:

  • Supply a PAC file to allow the plugin to automatically find the Proxy.
  • Supply the proxy information as Java parameters when activating the plugin. Use the following Java Proxy parameters:
    • http.proxyHost - IP address of the proxy
    • http.proxyPort - IP port used for connection
    • http.proxyUser - username for connection to proxy server
    • http.proxyPassword - password for connection to proxy server.
  • For proxies supporting SSL encryption, use the following Java Proxy parameters:
    • https.proxyHost - IP address of the proxy
    • https.proxyPort - IP port used for connection
    • https.proxyUser - username for connection to proxy server
    • https.proxyPassword - password for connection to proxy server.

For example:

gradle perfecto-android-inst -PconfigFileLocation="C:\temp\Espresso\ConfigFile.json" -Dhttp.proxyHost=10.0.0.100 -Dhttp.proxyPort=8800 -Dhttp.proxyUser=someUserName -Dhttp.proxyPassword=somePassword

Splitting the test

The Gradle Plugin supports the Android JUnitRunner's Sharding tests feature, running each test shard on a separate device. There are two options to run the split tests -

  • Splitting the test between all selected devices:
    • Select a number of devices in your Configuration File.
    • Configure the shard parameter to true (can be configured in either the Configuration File, build.gradle file, or on the command line).
    • Plugin will split the tests into a number of shards equal to the number of selected devices.
    • Each shard will run on a separate device and the summary report will identify the index of the shard for that device.
  • Run a specific shard
    • Select a single device in your Configuration file.
    • Configure the shard parameter to false (this turns-off the automatic splitting of the complete test).
    • Configure the numShards and shardIndex parameters in the instrumentationArgs configuration parameter (shown below in Configuration File format) to indicate the number of parts to split the test into and the index of the shard to run on the device.
    • The indicated shard will run on the selected device and the summary report will identify the index of the shard for that device.

      "shard" : false,
      "instrumentationArgs" : ["numShards"="5", "shardIndex"="3"] 

      Note: shardIndex value is in range [0..numShards-1] - index of first shard is 0, last shard is numShards-1

Screenshots

In addition to the screenshots that the plugin will take at the end of the test execution, based on the configuration parameters, the Espresso test methods may snap screenshots at key points during the test run. These screenshots will be retrieved by the plugin and included in the execution report (see below) from the following device folders:

  • /data/data/application-package-name/app_spoon-screenshots/test-class-name/test-method-name
  • /sdcard/app_spoon-screenshots/test-class-name/test-method-name

To take screenshots using the Spoon API - use the screenshot() method of the Spoon instance, as shown in the following examples:

Spoon.screenshot(activity, "initial_state");
/* Normal test code... */
Spoon.screenshot(activity, "after_login");

The tag specified may be used to identify and compare screenshots taken across multiple test runs.

Summary report on console

During execution

During the execution of the test methods, the plugin reports to the console:

  • Configuration settings resolution - the configuration file used and the list of the configuration settings for the test run.
  • Extensive progress notifications as the application starts and completes different steps
    • Notification of device allocation
    • Notification that the apk files have been installed to the devices.
    • Notification of video recording
    • Notification of test execution
    • Errors that may occur during any of these stages.
  • Completion status for each test method on each device.

The following is a sample of this report:

Acquiring device based on device management info {"platformName":"Android"}

Device: 0715F7A46C522232 Samsung Galaxy S6 Edge+: Operating on device {
	"deviceId":"0715F7A46C522232",
	"osVersion":"6.0.1",
	"screenResolution":"1440*2560",
	"location":"NA-US-BOS",
	"network":"Verizon-United States of America",
	"manufacturer":"Samsung",
	"model":"Galaxy S6 Edge+",
	"firmware":"zenltevzw-user 6.0.1 MMB29K G928VVRU2BPC2 release-keys",
	"distributor":"Generic",
	"description":"Stable PerfectoConnect",
	"selectionCriteria":{
		"platformName":"Android"},
	"os":"Android"} 
according to device details {
	"platformName":"Android",
	"index":1}

Device: 0715F7A46C522232 Samsung Galaxy S6 Edge+: Uploading APK to device server

Device: 0715F7A46C522232 Samsung Galaxy S6 Edge+: Completed uploading APK to device server

Device: 0715F7A46C522232 Samsung Galaxy S6 Edge+: Starting video recording

Device: 0715F7A46C522232 Samsung Galaxy S6 Edge+: Enabling instrumented test mode

Device: 0715F7A46C522232 Samsung Galaxy S6 Edge+: Trying to uninstall package com.example.perfecto.tipcalculator from device

Device: 0715F7A46C522232 Samsung Galaxy S6 Edge+: Uninstalled package com.example.perfecto.tipcalculator unsuccessfully

Device: 0715F7A46C522232 Samsung Galaxy S6 Edge+: Trying to uninstall package com.example.perfecto.tipcalculator.test from device

Device: 0715F7A46C522232 Samsung Galaxy S6 Edge+: Uninstalled package com.example.perfecto.tipcalculator.test unsuccessfully

Device: 0715F7A46C522232 Samsung Galaxy S6 Edge+: Installed APK successfully

Device: 0715F7A46C522232 Samsung Galaxy S6 Edge+: Installed test APK successfully

Device: 0715F7A46C522232 Samsung Galaxy S6 Edge+: Starting execution of package com.example.perfecto.tipcalculator.test with test runner android.support.test.runner.AndroidJUnitRunner

Device: 0715F7A46C522232 Samsung Galaxy S6 Edge+: Installing Android orchestrator

The following is a sample of an error message (failure to allocate device) as it appears in the report:

Failed to execute command handset open: Cannot open device VS9872C87734 for user myuser@mycompany.com, device is in use.
Command duration or timeout: 935 milliseconds
Build info: version: '2.53.1', revision: 'a36b8b1cd5757287168e54b817830adce9b0158d', time: '2016-06-30 19:26:09'
System info: host: 'nb-user-macosx.local', ip: '172.30.200.19', os.name: 'Mac OS X', os.arch: 'x86_64', os.version: '10.12.6', java.version: '1.8.0_131'
Driver info: org.openqa.selenium.remote.RemoteWebDriver

After execution

When execution completes on all devices, the plugin produces a summary report of the total execution status that includes:

  • Total summary - indicates whether there were any errors reported for the tests, and on how many devices the test ran successfully.
  • Per Device summary - for each device used - indicates
    • Final test run status
    • Device information
    • Number of test methods that reported success and failure on that device.
  • URL for the Execution report on the DigitalZoom Reporting server.

The following is a sample of this report:

----------------------------------------------------------------------------

TOTAL SUMMARY

Your task finished with errors.
FAILED TO RUN ON 0 devices
RAN SUCCESSFULLY ON 2 devices
----------------------------------------------------------------------------

DEVICE 1
Status: OK

Device:
03157DF34AD58C3A Samsung Galaxy S6

Results:
1 FAILED
5 PASSED
----------------------------------------------------------------------------

DEVICE 2
Status: OK

Device:
06157DF6AD8A2531 Samsung SM-G920W8

Results:
1 FAILED
5 PASSED
----------------------------------------------------------------------------

View the detailed report at: https://demo.reporting.perfectomobile.com/library?startExecutionTime[1]=lastMonth&tags[0]=5af27a82-54cc-405a-8c6e-fa46fcae874b

Finished flow execution

Summary report identifying shards run on different devices:

Execution report

As part of the summary report (see above) - the plugin provides the URL of the DigitalZoom STR for the execution run. Use the URL to connect to the reporting server and drill down to see the screenshots of the different test method executions and see what went wrong. In addition, the report will include attachments for:

  • Screenshots for each test run, based on the content of the folders specified above.
  • HAR files for each test - configure with the networkEmulation clause.
  • Vitals report for each test - configure with the vitalsMonitoring clause.

For more information on the STR see the Reporting overview.

Samples of plugin use

The Perfecto GitHub repository includes

The different samples demonstrate using the plugin for the following different modes/configurations:

  • defaultAndroidProjectSample - an Android project with the plugin Json config file located in the default place
  • localJarSample - an Android project configured to run with the Perfecto plugin jar file locally
  • pluginConfigurationSample - an Android project with the cloudURL/security token configured in the build.gradle file
  • remoteRunSample - how to run the plugin without the source code and only with apk's