SUT performance testing

Last updated: Mar 01, 2021 10:39

This article provides comprehensive information on Single User Performance Testing with Perfecto.

Implementation

Client-side performance testing is also called Single User Performance Testing (SUT) because in this case, no matter what the state of the servers is, it evaluates client-side KPIs like response time and device vitals.

SUT needs to take into account all of the factors that could impact the end-user experience across all tiers, including last-mile networks and client devices: network changes and signal degradation, changes in location, usage of onboard resources and sensors (that consume CPU, memory, and so on), background applications competing for device resources, and more. The following image shows external factors influencing application performance. So, simple functional testing is not enough.

All of these variables should be considered in test scenarios, and they should be automated because the time to test is limited. Service level agreements (SLA) for acceptable user experience should be set (for example, time to log in, time to accomplish a task) and measured in all scenarios.

The following example evaluates Etihad web application performance metrics.

The following transactions are under test: 

• Launch App: The time it takes to load the app
• Search Flights: The time it takes to list the available flights on the given itinerary

The Perfecto predefined function helps to simulate end-user conditions and offers the following abilities to extend coverage to real user conditions when testing user journeys or flows. The test includes the following sections:

1. End-user conditions that simulate network, location, background applications, and other properties

              //Set End User conditions
               //Throttle Network and Capture Traffic - In this example, HAR captured only for Android since Etihad IOS app expects trusted certificate
               if(platformName.equalsIgnoreCase("Android")) {
   	            Map<String, Object> pars = new HashMap<>();
   	            pars.put("profile", "4g_lte_good");
   	            pars.put("generateHarFile", "true");
   	            driver.executeScript("mobile:vnetwork:start", pars); 
               }
               else {
   	            Map<String, Object> pars = new HashMap<>();
   	            pars.put("profile", "4g_lte_good");
   	            driver.executeScript("mobile:vnetwork:start", pars); 
               }
               
               //Set location
   			Map<String, Object> params = new HashMap<>();
   			params.put("address", "Bangalore, India");
   			driver.executeScript("mobile:location:set", params); 
               
   			//Run background application
   			String backGroundApps = "YouTube,Messages,Maps,Calculator,Calendar";
   			 String[] bApps = backGroundApps.split(",");
   			 for(String i: bApps) {
   			    try {
   					Map<String, Object> params1 = new HashMap<>();
   					params1.put("name", i);
   					driver.executeScript("mobile:application:open", params1);
   				} catch (Exception e) {}		
   			 }

2. Points of interest that identify crucial application flows, such as AppLaunch and Search Flight

   reportTimer(driver, AppLaunchTime, 10000, "Checkpoint load time of App launch.", "AppLaunchTime");

3. Timer reporting ability to track timings and performance of the application
   An example might be how long it took for a page to load or a process to happen, and what that actually looks like for a user (so the UX or user experience on the device glass). In this test, we have chosen app launch time and flight search transactions.

   Perfecto provides page load time as User Experience Time (UX) with the help of visual functions (OCR and image analysis). In other words, the timer will wait for a specific text or image to actually appear (be rendered) on the screen. In our example, upon launching, the Etihad web timer will wait for the text  One Way to appear on the screen.

   The Perfecto page load time includes:

   Page Load Time = UX rendering time + (API process time + network time)

   For details, see Network traffic capture and analysis (HAR) | Performance testing.

   Note: The reference text or image should be the last component on the page that was loaded.

   //Launch Web application
   driver.get("https://www.etihad.com/en-us/book");
      ocrTextValidation(driver, "one Way");
      // Wind Tunnel: Measure UX timer 1 - Able to retrieve UX Timer value
      long AppLaunchTime = timerGet(driver, "ux");
      System.out.println("'Measured UX time for App launch time is: " + AppLaunchTime);
   
   private static void ocrTextValidation (RemoteWebDriver driver, String content) {	
      // verify that the correct page is displayed as result of signing in.
      Map<String, Object> params1 = new HashMap<>();
      // Check for the text that indicates that the sign in was successful
      params1.put("content", content);
      // allow up-to 30 seconds for the page to display
      params1.put("timeout", "30");
      // Wind Tunnel: Adding accurate timers to text checkpoint command
      params1.put("measurement", "accurate");
      params1.put("source", "camera");
      params1.put("analysis", "automatic");
      params1.put("threshold", "90");
      String resultString = (String) driver.executeScript("mobile:checkpoint:text", params1);
   }

4. A detailed report about the result of the script that is easy to understand and provides the following details:

   • Functional Validation: The feature is functional with detailed steps and a recorded video

      Click to view image

     
   • Performance validation, which includes:

     • Page Load Time: Measurement of responsiveness of the application’s rendered page load time retrieved by visual analysis

        Click to view image

       

     • Network Traffic: Report including a detailed (HAR) network traffic log. For details, see

     • Device Vitals: Recorded device vitals to provide visibility into network, CPU, and memory consumption during the execution of the test script

        Click to view image

       

A sample Java project is available here: https://github.com/PerfectoCode/Community-Samples/blob/master/JavaPerfectoUserExperience

The following video shows a sample execution.