Splunk | Configure executions with Quantum

Configure the TestNG XML for device redundancy when using Quantum to execute and feed monitor data to Splunk.

Anytime you schedule the execution of a "monitor," it should be done in a redundant manner. Redundancy, in this case, means you should always execute the test case against at least two different devices of the same type or two different browser instances. The reason behind redundancy is to get rid of any potential flakiness due to environmental issues and to ensure you are getting legitimate results.

Important: This document includes references to a third-party product, Splunk. The user interface and usage of third-party products are subject to change without notice. For the latest published information about Splunk, see https://docs.splunk.com/Documentation.

Duplicate the test node

The way to schedule redundant executions is to add two separate "test" nodes to the TestNG XML file. The tests should have the exact same configurations with the exception of the device selection capabilities.

Copy 
<test name="Android Amazon Search Test" enabled="true">
    <parameter name="perfecto.env.resources" value="src/main/resources/android" />
    <parameter name="perfecto.capabilities.deviceName" value="04157DF43A656B1A" />
    <parameter name="perfecto.capabilities.browserName" value="mobileOS"></parameter>
    <groups>
        <run>
            <include name="@amazonSearch" />
        </run>
    </groups>
    <classes>
        <class name="com.qmetry.qaf.automation.step.client.gherkin.GherkinScenarioFactory" />
    </classes>
</test> 
<test name="iOS Amazon Search Test" enabled="true">
    <parameter name="perfecto.env.resources" value="src/main/resources/ios" />
    <parameter name="perfecto.capabilities.deviceName" value="1C3B401545D2CDBEC9D323460D914AD7319F31D9" />
    <parameter name="perfecto.capabilities.browserName" value="mobileOS"></parameter>
    <groups>
        <run>
            <include name="@amazonSearch" />
        </run>
    </groups>
    <classes>
        <class name="com.qmetry.qaf.automation.step.client.gherkin.GherkinScenarioFactory" />
    </classes>
</test>

Device selection capabilities

The example XML code includes two separate hard-coded device IDs. However, you could simply add the capability to generically pick model types or platform types/versions, as shown in the examples

Copy 
<parameter name="perfecto.capabilities.model" value="iPhone-.*" />

or

Copy 
<parameter name="perfecto.capabilities.platformName" value="iOS" /><parameter name="perfecto.capabilities.platformVersion" value="10.*|10.*.*" />

Read more on device selection capabilities here.

Parallel execution

Next, you can execute the test in parallel so that you are capturing the results at the exact same time (or close to it), to do this we need to enable parallel executions in the XML. To enable parallel execution of the test cases we simply add thread-count="2" (in the case that we are only using two devices for redundancy if you are using more modify this number to match the number of redundant tests you have set) and add parallel="tests" to the suite node as shown at the top of the below XML.

Copy 
<suite name="Appium Demo Suite" verbose="0" thread-count="2" parallel="tests">
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
    <listeners>
        <listener class-name="com.quantum.listerners.QuantumReportiumListener" />
        <listener class-name="com.quantum.listerners.TestListener" />
    </listeners>
    <parameter name="driver.name" value="perfectoRemoteDriver" />
    <test name="Android Amazon Search Test" enabled="true">
        <parameter name="perfecto.env.resources" value="src/main/resources/android" />
        <parameter name="perfecto.capabilities.deviceName" value="04157DF43A656B1A" />
        <parameter name="perfecto.capabilities.browserName" value="mobileOS"></parameter>
        <groups>
            <run>
                <include name="@amazonSearch" />
            </run>
        </groups>
        <classes>
            <class name="com.qmetry.qaf.automation.step.client.gherkin.GherkinScenarioFactory" />
        </classes>
    </test>
    <test name="iOS Amazon Search Test" enabled="true">
        <parameter name="perfecto.env.resources" value="src/main/resources/ios" />
        <parameter name="perfecto.capabilities.deviceName" value="1C3B401545D2CDBEC9D323460D914AD7319F31D9" />
        <parameter name="perfecto.capabilities.browserName" value="mobileOS"></parameter>
        <groups>
            <run>
                <include name="@amazonSearch" />
            </run>
        </groups>
        <classes>
            <class name="com.qmetry.qaf.automation.step.client.gherkin.GherkinScenarioFactory" />
        </classes>
    </test>