Page tree
Skip to end of metadata
Go to start of metadata

Last updated: Oct 17, 2018 12:33

Preparing the Application Files

Executing XCUITest and XCTest tests involve two application files:

  • The application ipa file - this includes the XCTest unit tests.
  • The UI Runner ipa file - this includes the XCUITest UI test methods.

Use the following steps to create the files as ipa files :

  • Step 1: Change scheme destination to Generic IOS device.

  • Step 2: Click Product > Build For > Testing. Once this is complete, open up the Organiser and select the latest version.

  • Step 3: Right click on the generated .app file in the Project Navigator (in the Products folder) and select Show in Finder

  • Step 4: Finder will display the path to the Products folder, and sill show two .app packages -

    • The application package

    • An app package for the UI Tests named *UITests-Runner.

There are two options for using these files:

    1. Supply the path to the application package path to this file as the testAppPath configuration parameter. The Gradle plugin will convert to .ipa file.
    2. Use the following steps to convert each of the files to ipa files:
      • Step 1: Create an empty "Payload" (case sensitive) folder and copy the .app bundle to this folder.

        Note: If the Payload folder already exists, delete it and create a new empty folder.

      • Step 2: Right click on the Payload folder and Compress it to a zip file.
      • Step 3: Rename the .zip file to <testName>.ipa

Activating the Plugin Task

To activate the plugin - Open a command-line window in the project's folder and run the perfecto-xctest task in gradle:

gradle perfecto-xctest <configuration parameters>

This will

  • Select the devices as specified in the Device selection parameters of the configuration file [or a random device if no specification provided],
  • Install the application and test ipa files onto the device
  • Run the test methods (based on the configuration parameters)
  • Send output to the Gradle console
  • Generate an execution Single Test Report that can be viewed in the DigitalZoom Reporting interface.

The command line parameters can set any of the configuration parameters except for device selection parameters. For more information on the configuration parameters read this page.

Running over a proxy connection

If the Perfecto Lab is connected via a proxy, you will need to supply the proxy information as Java parameters when activating the plugin. Use the following Java Proxy parameters:

  • http.proxyHost - IP address of the proxy
  • http.proxyPort - IP port used for connection
  • http.proxyUser - username for connection to proxy server
  • http.proxyPassword - password for connection to proxy server.

For proxies supporting SSL encryption, use the following Java Proxy parameters:

  • https.proxyHost - IP address of the proxy
  • https.proxyPort - IP port used for connection
  • https.proxyUser - username for connection to proxy server
  • https.proxyPassword - password for connection to proxy server.

For example:

gradle perfecto-xctest -PconfigFileLocation="C:\temp\Espresso\ConfigFile.json" -Dhttp.proxyHost=10.0.0.100 -Dhttp.proxyPort=8800 -Dhttp.proxyUser=someUserName -Dhttp.proxyPassword=somePassword

Screenshots (XCTest methods only)

XCTest test methods may snap screenshots at key points during the test run. These screenshots will be retrieved by the plugin and included in the execution report. The plugin supports screenshots generated by either:

  • KIF system KIFSystemTestActor class
  • EarlGrey screenshots

KIF Screenshots

To take screenshots using the KIFSystemTestActor captureScreenshotWithDescription() method:

  • Use the instrumentationArgs configuration parameter to indicate using KIF for screenshots

    "instrumentationArgs" : ["KIF"]

    This will preset the KIF_SCREENSHOTS environment variable for the test run.

  • Supply a description parameter that includes the test's className and testMethod in the following format:
    "testClass#testMethod" (full description may include a prefix or suffix to this string). This format allows to associate the screenshot to the correct test method

EarlGrey Screenshots

To take screenshots using the EarlGrey GREYScreenshotUtil class:

  • Use the instrumentationArgs configuration parameter to indicate using EarlGrey for screenshots

    "instrumentationArgs" : ["EARLGREY"]
  • The screenshot should be in a "png" file and the name should include a string that includes the test's className and testMethod in the following format:
    "testClass#testMethod.png" (full description may include a prefix or suffix to this string). This format allows to associate the screenshot to the correct test method.
  • The following XCode snippet provides an example of creating the screenshot:

            let confpath = GREYConfiguration.sharedInstance().value(forConfigKey: kGREYConfigKeyArtifactsDirLocation) as! String
            var screenshot = GREYScreenshotUtil.takeScreenshotForAppStore()
            var filename = String(describing: EarlGreySimpleIosAppForTestingTests.self) + "#" + #function + ".png"
            GREYScreenshotUtil.saveImage(asPNG: screenshot, toFile: filename, inDirectory: confpath)
            EarlGrey.select(elementWithMatcher: grey_text("EarlGrey Tests")).assert(grey_sufficientlyVisible())

Summary report on console

During execution

During the execution of the test methods, the plugin reports to the console:

  • Configuration settings resolution - the configuration file used and the list of the configuration settings for the test run.
  • Progress notifications as the test is configured, installed, and executed including notifications of the start and completion of:
    • Allocating the device
    • Resigning the application
    • Installing the application and UI runner files
    • Executing the test
    • Connecting the video recording
  • Completion status for each test method on each device.

The following is a sample of this report:

Parsing configuration file: configFile.json

Parsed configuration file configFile.json successfully

Configuration:
cloudURL='YourLab.perfectomobile.com'
appName='app'
tags='[test_1, test2]'
projectName='IOS-XCUITest'
projectVersion='v10.9-test'
jobName='some_job'
jobNumber='2'
branch='master'
failBuildOnFailure='false'
debug='false'
devices='[DeviceDetails{deviceName='047CEB33AF9306ED58A002DAA400DF726C38AAC1', platformName='IOS', platformVersion='null', manufacturer='null', model='null', resolution='null', network='null', location='null', description='null', selectionCriteria='null', index='1'}]'
acquireDeviceRetryNumber='0'
acquireDeviceRetryInterval='30'
postExecution='PostExecutionDetails{uninstall=false}'
preCleanUp='false'
resign='true'
takeScreenshotOnTestEnd='false'
takeScreenshotOnTestFailure='false'
shard='false'
ipaPath='/Users/.../root/xcodeSampleApp/ipas/SimpleAppForTesting.ipa'
testIpaPath='/Users/.../root/xcodeSampleApp/ipas/Payload.ipa'
runUiTests='true'
runUnitTests='true'


Device: 00083F5BF9243A97E19FA80FAD9D8649653F6819 Apple iPhone-7 Plus: Operating on device {
	"deviceId":"00083F5BF9243A97E19FA80FAD9D8649653F6819",
	"osVersion":"11.3",
	"screenResolution":"1242*2208",
	"location":"NA-US-PHX",
	"manufacturer":"Apple",
	"model":"iPhone-7 Plus",
	"firmware":"11.3",
	"distributor":"MacCam11",
	"selectionCriteria":{
		"platformVersion":"^[9]\\.[3-9].*|^[1-9][0-9].*",
		"platformName":"IOS"},
	"os":"iOS"} 
according to device details {
	"platformName":"IOS",
	"platformVersion":"^[9]\\.[3-9].*|^[1-9][0-9].*",
	"automationName":"PerfectoMobile",
	"index":2}

Device: 00083F5BF9243A97E19FA80FAD9D8649653F6819 Apple iPhone-7 Plus: Resigning application

Device: 00083F5BF9243A97E19FA80FAD9D8649653F6819 Apple iPhone-7 Plus: Completed resigning application

Device: 00083F5BF9243A97E19FA80FAD9D8649653F6819 Apple iPhone-7 Plus: Uploading IPA to device server

Device: 00083F5BF9243A97E19FA80FAD9D8649653F6819 Apple iPhone-7 Plus: Completed uploading IPA to device server

Device: 00083F5BF9243A97E19FA80FAD9D8649653F6819 Apple iPhone-7 Plus: Starting video recording

Device: 00083F5BF9243A97E19FA80FAD9D8649653F6819 Apple iPhone-7 Plus: Enabling instrumented test mode

Device: 00083F5BF9243A97E19FA80FAD9D8649653F6819 Apple iPhone-7 Plus: Uninstalling IPA

Device: 00083F5BF9243A97E19FA80FAD9D8649653F6819 Apple iPhone-7 Plus: Uninstalling test bundle

Device: 00083F5BF9243A97E19FA80FAD9D8649653F6819 Apple iPhone-7 Plus: Installing ipa SimpleAppForTesting.ipa

Device: 00083F5BF9243A97E19FA80FAD9D8649653F6819 Apple iPhone-7 Plus: IPA  C:\Windows\TEMP\ITES\4b918fc2-f1a1-4d7c-9241-1730cfe471cb\00083F5BF9243A97E19FA80FAD9D8649653F6819\2\Apps\SimpleAppForTesting.ipa Installed

Device: 00083F5BF9243A97E19FA80FAD9D8649653F6819 Apple iPhone-7 Plus: Installing ipa SimpleAppForTesting-Runner.ipa

Device: 00083F5BF9243A97E19FA80FAD9D8649653F6819 Apple iPhone-7 Plus: IPA  C:\Windows\TEMP\ITES\4b918fc2-f1a1-4d7c-9241-1730cfe471cb\00083F5BF9243A97E19FA80FAD9D8649653F6819\2\Apps\SimpleAppForTesting-Runner.ipa Installed

Device: 00083F5BF9243A97E19FA80FAD9D8649653F6819 Apple iPhone-7 Plus: Starting execution of bundle id com.igor.SimpleAppForTesting

The following is a sample of an error message (failure to allocate device) as it appears in the report:

Failed to execute command handset open: Cannot open device FA6BR0304878 for user myuser@mycompany.com, device is in use.
Command duration or timeout: 1.20 seconds
Build info: version: '2.53.1', revision: 'a36b8b1cd5757287168e54b817830adce9b0158d', time: '2016-06-30 19:26:09'
System info: host: 'nb-user-macosx.local', ip: '172.30.200.19', os.name: 'Mac OS X', os.arch: 'x86_64', os.version: '10.12.6', java.version: '1.8.0_131'
Driver info: org.openqa.selenium.remote.RemoteWebDriver

After execution

When execution completes on all devices, the plugin produces a summary report of the total execution status that includes:

  • Total summary - indicates whether there were any errors reported for the tests, and on how many devices the test ran successfully.
  • Per Device summary - for each device used - indicates
    • Final test run status
    • Device information
    • Number of test methods that reported success and failure on that device.
  • URL for the Execution report on the DigitalZoom Reporting server.

The following is a sample of this report:

----------------------------------------------------------------------------

TOTAL SUMMARY

Your task finished with errors.
FAILED TO RUN ON 0 devices
RAN SUCCESSFULLY ON 1 device
----------------------------------------------------------------------------

DEVICE 1
Status: OK

Device:
047CEB33AF9306ED58A002DAA400DF726C38AAC1 Apple iPhone-6

Results:
3 FAILED
7 PASSED
----------------------------------------------------------------------------

View the detailed report at:
https://YourLab.reporting.perfectomobile.com/library?tags[0]=3d2c6e96-d54e-462e-bae0-8f350f509adc

Finished flow execution

Execution report

As part of the summary report (see above) - the plugin provides the URL of the DigitalZoom STR for the execution run. Use the URL to connect to the reporting server and drill down to see the screenshots of the different test method executions and see what went wrong. For more information on the STR see the Reporting overview.

Samples of plugin use

The Perfecto GitHub repository includes

The different samples demonstrate using the plugin for the following different modes/configurations:

  • defaultAndroidProjectSample - an Android project with the plugin Json config file located in the default place
  • localJarSample - an Android project configured to run with the Perfecto plugin jar file locally
  • pluginConfigurationSample - an Android project with the cloudURL/security token configured in the build.gradle file
  • remoteRunSample - how to run the plugin without the source code and only with apk's