Perfecto allows you to gain continuous insight into your application behavior across builds, devices and more, in the dashboard of your choice.
Many enterprises today use separate tools, processes and teams to gain insight on the application health before and in production. This approach is sub-optimal for those organizations wishing to become agile while maintaining high-quality applications. The following article will describe the Perfecto approach in more details.
When you are responsible for the quality of a mobile application, you're usually responsible for 3 versions of the app (at a minimum):
- The one in production: let's call it V5
- The one that's almost done in the sprint: let's call it V6
- The one that's just starting: let's call it V7
The reality of separate teams, tools and process creates roadblocks at many phases, including:
- The next build (V6) is ready to launch, but the operations team and tool aren't. The team is unable to launch until the operations tool and team are ready.
- The app in production, using the production build (V5) is now down. Users are upset and the business is impacted. The operations team need help because the issue is in the application logic, not a server that went down. However, using a separate tool and process, the dev team needs to learn what the issue actually is. The dev team is disrupted (recurringly), and it takes much longer to troubleshoot issues.
- Lastly, when product management wants to plan the next set of build features (V7), they want to include enhancements to app weaknesses. Having a separate tool and process, they do not have the visibility they need into app issues in production.
A mutual tool, team, and process to own the application behavior across versions, devices, personas (and any other parameter you want) could come very handy. Especially if it involves little effort. Learn how
On this page:
Architecture
The architecture is based on the same tools and processes used by the testing team:
- A scheduler: Jenkins, Visual Studio or any of your choice
- A script repository: GitHub, SVN, or similar
- Perfecto's continuous testing platform, made of devices and browsers
- Your very own dashboard/business intelligence tool, in this example, Splunk
It is worth noting that it is not our intention to flood your dashboard/Splunk with all of Perfecto data. We produce a lot of data, some of it is rich media (videos, screenshots etc.). All of which you can, of course, download from Perfecto into your repository. Rather, we bring to your dashboard the essential data to allow you to trend, alert etc.:
- Script meta data: the script name/version, details of the device under test, persona etc.
- Script execution data: script and step-level pass/fail, step names, user experience timing etc.
To read more about the data sent to Splunk, see the article Splunk | Perfecto-Splunk data schema.
Once you come to the step of drilling into a single test report, every point in the dashboard would click over to Perfecto's report, where all of that data exist.
Back to the architecture: If you have 1-3, there are only two additions you need to make:
- Getting the data over to Splunk
- Setting Splunk to show the data you want, dashboards, alerts, email reports etc.
Code
This has been revised as the library has been updated to support both Splunk Cloud and Splunk Enterprise products.
Splunk support
This project is supported by the Splunk HTTP Event Collector available in Splunk 6.3 and above. Below you will find the details on how to configure the connector for use.
Splunk | Create a custom data source
To allow for the extracting of fields automatically by the HEC we must create a new Data Source, please follow the directions here to do so.
Splunk | Create an index
To learn how to create an index, please read here.
Splunk | HEC configuration
-
Splunk Cloud Customers - Contact Splunk support to have a public facing HEC created for you. You will need to supply them the index and source type we created above when you submit the request.
-
Splunk Enterprise - Read here on how to configure your Splunk instance to work with the library.
-
GitHub Project - Splunk Connector GitHub Project can be found here.
Clone the Github code into Eclipse
Take a look at this Stack Overflow Answer on how to clone the project into Eclipse.
Usage
Either export the project as a jar or add the project as a dependency to your existing framework.
The library is a Maven project but if you have issues in your environment accessing Maven then read below for the required dependencies.
The project only has two dependencies: Gson 2.5 you can find here and Jackson XML 2.8.1 which you can find here. Other versions of the dependencies may work but have not been tested.
Add the PerfectoSplunk jar to your Java project
Take a look at this Stack Overflow Answer to learn how to import a jar into your project.
Usage
You may want to review this article before proceeding so you understand where and how it may make sense to implement the reporting collector into your framework: TestNG Report Collection
Available options
-
Reporting - Submit a Splunk event for EACH test case. This option is meant for usage in a standard automation framework.
-
ReportingCollector - Collect multiple test executions and submit to Splunk as a single event sorted by Primary and Secondary records based on Pass and Fail of the test case and the SLA status. This option is meant for usage in a monitoring/performance framework.
Create the instance
Default values
| Product | Parameter | Value |
|---|---|---|
| Splunk Enterprise | Hostname | Standard host name or IP of your internal server |
| Port | 8088 (configurable in the Http Event Collector configuration) | |
| Scheme | http (configurable in the Http Event Collector configuration) | |
| Splunk Cloud | Hostname | Your cloud hostname with http-inputs appended to the beginning. For example, if my cloud is myCloud.SplunkCloud.com, my host name will actually be http-inputs-myCloud.SplunkCloud.com. |
| Port | 443 | |
| Scheme | https |
To start you must perform the below code to generate the reporting instance. Keep in mind the SLA value is a global value which will impact all transactions.
// Call this at the start of test to set the reporting class and
// define splunk details
// Params
// @1 long SLA in milliseconds - this will define pass/fail threshold for // steps based on response time
// @2 String Splunk scheme http or https
// @3 String Splunk host
// @4 String Splunk port
// @5 String Splunk token if required
// @6 Proxy Object if desired
SplunkReportingCollector reporting = ReportingCollectorFactory.createInstance(splunkSLA, splunkScheme, splunkHost,splunkPort, splunkToken);
//Pass the created Factory to the setReporting method
//Params
//@1 The reporting instance created above
ReportingCollectorFactory.setReporting(reporting);The code above is showing how to create the ReportingCollector instance however if you want to use the Reporting instance instead use the below.
// Call this at the start of test to set the reporting class and
// define splunk details
// Params
// @1 long SLA in milliseconds - this will define pass/fail threshold for // steps based on response time
// @2 String Splunk scheme http or https
// @3 String Splunk host
// @4 String Splunk port
// @5 String Splunk token if required
// @6 Proxy Object if desired
SplunkReporting reporting = ReportingFactory.createInstance(splunkSLA, splunkScheme, splunkHost,splunkPort, splunkToken);
//Pass the created Factory to the setReporting method
//Params
//@1 The reporting instance created above
ReportingFactory.setReporting(reporting);Access the instance
Now to use the instance we need to use the code ReportingCollectorFactory.getCollector(). The below method shortcuts the code so you can simply call getCollector() to retrieve the instance.
//Returns the reportingCollector class for use
public SplunkReportingCollector getCollector()
{
return ReportingCollectorFactory.getCollector();
}The above code is specific to the ReportingCollector if you want to just use Reporting use the below code instead.
//Returns the reporting class for use
public SplunkReporting getCollector()
{
return ReportingFactory.getReporting();
}Test start and end times
Test execution start and end have embedded methods in reporting to allow for easy calculation of execution duration and proper formatting. The start and end times as well as the duration will be stored in the Json string sent to Splunk.
//Sets the start time of the test
getCollector().testExecutionStart();
//Sets the end time of the test
getCollector().testExecutionEnd();Track transactions
You also have the ability to track individual steps. You need to wrap the steps you want tracked as part of the transaction in the startTransaction and endTransaction commands. The transaction time is what is used on to determine if you met your SLA specified when initializing the reporting instance.
//sets the start of a transaction
//Params
//@1 short description of the step
//@2 a more detailed description of the step
getCollector().startTransaction(step, "goto Amazon.com");
//sets the end of a transaction
//Params
//parameter 1 is the same short description as the start transaction
//parameter 2 is the Perfecto timer you retrieved to validate against the SLA
getCollector().endTransaction(step, lib.getUXTimer());
//sets the end of a transaction
//Params
//@1 allows you to set the SLA in miliseconds directly in the method instead //of needing to call setSLA on another line, as before this is a global change
//@2 is the same short description as the start transaction
//@3 is the Perfecto timer you retrieved to validate against the SLA
getCollector().endTransaction(1000, step, lib.getUXTimer());Change the SLA at any time
You also have the option of changing your SLA any time by calling the setSLA method, this globally changes the SLA for all transactions moving forward.
//sets global SLA for the reporter
//Params
//@1 allows you to set the SLA in miliseconds
getCollector().setSLA(5000);Custom parameters
To add additional custom reporting parameters to your JSON, you can utilize the following:
//Put any value into the reporting json by passing
//Params
//@1 parameter name
//@2 parameter value
getCollector().reporting.put("testStatus", result);Submit to the collector (not required if only using Perfecto Smart Reporting)
When your test case completes, you need to submit the test details to the collector for storage. When you are ready, call the below code. This is NOT required for Reporting, only the ReportingCollector.
//Submits the test case to the collector
//Params
//@1 test name
getCollector().submitReporting("testName");Commit to Splunk
When you are ready to submit your data to Splunk, you call the below code for the ReportingCollector.
//Submits the data to splunk
//return type is String and contains the Json which was written to Splunk
getCollector().commitSplunk();Or the below code for Reporting
// Submits the data to splunk
// Params
// @1 High level test name - first node in JSON
// @2 test case name
// return type is String and contains the Json which was written to
// Splunk
String value = getCollector().commitSplunk("Perfecto", "testname");Proxy configuration
The library also supports proxy servers. If you need to configure a proxy for the library then before calling createInstance add the below code and modify for your environment.
Proxy proxy = new
Proxy(Proxy.Type.HTTP, new InetSocketAddress("yourHost", 8080));If your proxy also requires authentication then also place the below after the Proxy object and modify it for your environment as well.
Proxy proxy = new
Proxy(Proxy.Type.HTTP, new InetSocketAddress("yourHost", 8080));
Authenticator authenticator =
new Authenticator() {
public PasswordAuthentication getPasswordAuthentication() {
return (new PasswordAuthentication("user",
"password".toCharArray()));
}
};
Authenticator.setDefault(authenticator)Now you can pass the proxy object to the createInstance method as the final parameter.
SplunkReportingCollector reporting = ReportingCollectorFactory.createInstance(1000, "splunkHostName", 8089, "https", "", "username","password", proxy);
ReportingCollectorFactory.setReporting(reporting);Find sample projects using the ReportingCollector method here.
Sample project reporting
To run the below project import the PerfectoSplunk jar and modify the testSuite.xml to match your Splunk and Perfecto parameters.
Find a sample project using the Reporting method, click here.
Example of Reporting | Splunk Payload
{ [-]
Perfecto: { [-]
className: tests.TestSystem
device: 02157DF2AB41E718
executionID: user@perfectomobile.com_RemoteWebDriver_16-08-20_05_35_44_63278
hostName: WIN-CQV51E0V7CF
methodName: OrderBook
methods: { [-]
OrderBook: { [-]
Steps: [ [-]
{ [-]
step: Step_1
stepDescription: goto Amazon.com
stepEndTimestamp: Aug 20, 2016 1:36:04 AM
stepStartTimestamp: Aug 20, 2016 1:35:56 AM
stepStatus: Pass
stepTimer: 94
}
{ [-]
step: Step_2
stepDescription: Enter book into search box
stepEndTimestamp: Aug 20, 2016 1:36:33 AM
stepStartTimestamp: Aug 20, 2016 1:36:04 AM
stepStatus: Pass
stepTimer: 0
}
]
}
}
model: Galaxy S6
os: Android
perfectoReport: https://demo.perfectomobile.com/nexperience/Report.html?reportId=SYSTEM%3Adesigns%2Freport&key=PUBLIC:RemoteWebDriver/RemoteWebDriver_16-08-20_05_35_44_63278%2Exml&liveUrl=rtmp%3A%2F%2Fdemo%2Eperfectomobile%2Ecom%2Fengine&appUrl=https%3A%2F%2Fdemo%2Eperfectomobile%2Ecom%2Fnexperience
performanceStatus: Pass
reportKey: PUBLIC:RemoteWebDriver/RemoteWebDriver_16-08-20_05_35_44_63278.xml
seleniumHost: demo.perfectomobile.com
testExecutionDuration: 36
testExecutionEnd: Aug 20, 2016 1:36:33 AM
testExecutionStart: Aug 20, 2016 1:35:56 AM
testName: Test Galaxy S6 4G
testStatus: Pass
threadNo: TestNG 12
windTunnelReport: https://demo.perfectomobile.com/nexperience/singletest/report/?reportRepositoryKey=PUBLIC:RemoteWebDriver/RemoteWebDriver_16-08-20_05_35_44_63278.xml&ownerId=user@perfectomobile.com&sharingCode=d6731762-65b7-4eae-a51b-4ab85424ba75 Example of ReportiungCollector | Splunk Payload
{ [-]
Primary: { [-]
className: tests.TestSystem
device: 21FA87D12F5451B436E9918C6FA1800918696394
executionID: user@perfectomobile.com_RemoteWebDriver_16-08-22_19_41_46_75107
hostName: WIN-CQV51E0V7CF
methodName: OrderBook
methods: { [-]
OrderBook: { [-]
Steps: [ [-]
{ [-]
step: Step_1
stepDescription: goto Amazon.com
stepEndTimestamp: Aug 22, 2016 3:42:02 PM
stepStartTimestamp: Aug 22, 2016 3:41:53 PM
stepStatus: Pass
stepTimer: 78
}
{ [-]
step: Step_2
stepDescription: Enter book into search box
stepEndTimestamp: Aug 22, 2016 3:42:34 PM
stepStartTimestamp: Aug 22, 2016 3:42:02 PM
stepStatus: Pass
stepTimer: 0
}
]
}
}
model: iPhone-6
os: iOS
perfectoReport: https://demo.perfectomobile.com/nexperience/Report.html?reportId=SYSTEM%3Adesigns%2Freport&key=PUBLIC:RemoteWebDriver/RemoteWebDriver_16-08- 22_19_41_46_75107%2Exml&liveUrl=rtmp%3A%2F%2Fdemo%2Eperfectomobile%2Ecom%2Fengine&appUrl=https%3A%2F%2Fdemo%2Eperfectomobile%2Ecom%2Fnexperience
performanceStatus: Pass
reportKey: PUBLIC:RemoteWebDriver/RemoteWebDriver_16-08-22_19_41_46_75107.xml
seleniumHost: demo.perfectomobile.com
testExecutionDuration: 41
testExecutionEnd: Aug 22, 2016 3:42:34 PM
testExecutionStart: Aug 22, 2016 3:41:53 PM
testName: Test iPhone-6
testStatus: Pass
threadNo: TestNG 13
windTunnelReport: https://demo.perfectomobile.com/nexperience/singletest/report/?reportRepositoryKey=PUBLIC:RemoteWebDriver/RemoteWebDriver_16-08-22_19_41_46_75107.xml&ownerId=user@perfectomobile.com&sharingCode=50f5a79b-1394-4547-8fb4-407590e2173f
}
Secondary1: { [-]
className: tests.TestSystem
device: 1215FC1949B53804
executionID: user@perfectomobile.com_RemoteWebDriver_16-08-22_19_41_46_75106
hostName: WIN-CQV51E0V7CF
methodName: OrderBook
methods: { [-]
OrderBook: { [-]
Steps: [ [-]
{ [-]
step: Step_1
stepDescription: goto Amazon.com
stepEndTimestamp: Aug 22, 2016 3:42:08 PM
stepStartTimestamp: Aug 22, 2016 3:42:01 PM
stepStatus: Pass
stepTimer: 47
}
{ [-]
step: Step_2
stepDescription: Enter book into search box
stepEndTimestamp: Aug 22, 2016 3:42:39 PM
stepStartTimestamp: Aug 22, 2016 3:42:08 PM
stepStatus: Pass
stepTimer: 0
}
]
}
}
model: Galaxy S6
os: Android
perfectoReport: https://demo.perfectomobile.com/nexperience/Report.html?reportId=SYSTEM%3Adesigns%2Freport&key=PUBLIC:RemoteWebDriver/RemoteWebDriver_16-08-22_19_41_46_75106%2Exml&liveUrl=rtmp%3A%2F%2Fdemo%2Eperfectomobile%2Ecom%2Fengine&appUrl=https%3A%2F%2Fdemo%2Eperfectomobile%2Ecom%2Fnexperience
performanceStatus: Pass
reportKey: PUBLIC:RemoteWebDriver/RemoteWebDriver_16-08-22_19_41_46_75106.xml
seleniumHost: demo.perfectomobile.com
testExecutionDuration: 38
testExecutionEnd: Aug 22, 2016 3:42:39 PM
testExecutionStart: Aug 22, 2016 3:42:01 PM
testName: Test Galaxy S6
testStatus: Pass
threadNo: TestNG 12
windTunnelReport: https://demo.perfectomobile.com/nexperience/singletest/report/?reportRepositoryKey=PUBLIC:RemoteWebDriver/RemoteWebDriver_16-08-22_19_41_46_75106.xml&ownerId=user@perfectomobile.com&sharingCode=2acb3881-b56d-42f2-b756-f1996f88be5a With your data into Splunk, you can create customer dashboards and alerts to visualize and monitor your data.
