Implement failure reasons with Quantum 1.20

Failure Reasons Quantum version 1.20, the new feature of managing failure reasons in Perfecto Smart Reporting, is now supported and integrated. Quantum 1.20 supports adding failure reasons based on test failure exceptions. For example, you can tag all failures with a specific exception message (Example: com.qmetry.qaf.automation.step.StepInvocationException: java.lang.Exception) with a failure reason that was previously added through the admin option in Perfecto (Example: Custom Failure Reason).

Learn how you can leverage this functionality.

If you use Quantum 1.19, watch this video to see how you can add failure reasons.

Prerequisites

To be able to use the Failure Reasons feature, along with custom fields and tags in reports, make sure you have upgraded to Quantum version 1.20. If you are still on an older version of Quantum, see Quantum 1.20 release notes for upgrade information. Here are some details about which lines in your POM you need to edit:

Add failure reasons to reports

To configure the Failure Reason feature, add a JSON file with the name failureReasons.json at the following location: src/main/resources/failureReasons.json. If you want to use a custom path to define this file, then configure this key in the application properties with your custom path, as follows: 

#Use this key to configure the path of the failureReasons.json file. By default the path will be src/main/resources/failureReasons.json
failureReasonConfig=src/main/failureReasons.json

The failure reason for the exception message mapping can be one-to-many. You can define this configuration in the failureReasons.json config file. You can also attach a list of custom fields and tags along with the failure reason. Following is a sample failureReasons.json file.

[  
   {  
      "CustomError": "Custom Failure Reason",
      "StackTraceErrors":[  
         "org.openqa.selenium.TimeoutException: Expected condition failed: waiting for presence of element located by:",
         "org.openqa.selenium.WebDriverException: Get by url is not supported.",
      ],
      "CustomFields" :  
         [
            "version-11",
            "Test-ok"         ],
      "Tags" : ["Sanity", "Nightly"]
   },
   {  
      "CustomError": "Element not found",
      "StackTraceErrors":[  
         "<<Exception Message 1>",
         "<<Exception Message 2>"      ]
   }
] 

You need to maintain the JSON syntax as shown here. The parameter "Element not found" is a failure reason that exists by default in Perfecto Smart Reporting.

The array of exception messages linked to these failure reasons can be a substring of the exception message that we get from our test failure, as shown in the following image for the "Custom Failure Reason" example shown in the JSON file. The exception message string "com.qmetry.qaf.automation.step.StepInvocationException: java.lang.Exception" was a part of the test failure exception. Based on our configurations, Quantum added the "Custom Failure Reason" tag to this test result.

There is no limit to the number of exceptions that you can link with one failure reason. You can use the array syntax to add more than one exception message, as shown in the sample for the "Element not found" failure reason.

Custom fields are an array of key/value pairs with a delimiter (default "-"). If you want to use a custom delimiter, then configure this key in the application properties with your custom path: 

#Use this key to configure the custom field delimiter. By default the delimiter will be "-"custom.field.delimiter=-

Valid exceptions

You can only add custom failure reasons for exceptions that are thrown as a result of a failing command. For example:

Some commands do not throw an exception when they fail. For example, some of the ElementNotFound errors you see in commands that are generated from methods like verifyPresent do not generate exceptions. If this case, Quantum generates a stack trace on background to provide more details on why the command failed. You cannot use these exceptionsfor the creation of a Custom Failure Reason. You can recognize them by the specific sentence you will see on the second line of the stack trace:

There was some validation failure in the scenario which did not provide any throwable object

You can also provide custom exceptions thrown during test executions for specific conditions. For example, IF a specific element is not found, throw an exception (same example: verifyPresent does not throw an exception, so you may want to catch it in an IF statement and throw your own exception or use Log Assert to fail the test and use that stack trace to generate a Custom Failure Reason).

Here is sample code on how to create such a scenario in your steps, where the custom exception "Backend issue" is the one that you have to add in your failureReasons.json:

    @Given("^I am on Google Search Page$")
    public void I_am_on_Google_Search_Page() throws Exception {
        new WebDriverTestBase().getDriver().get("http://www.google.com/");
//        Client project code which will produce data base connection error when the backend systems are down.
        throw new Exception("Backend issue");
    }

[
   { 
      "CustomError": "Custom Failure Reason",
      "StackTraceErrors":[
         "Stack Trace: org.openqa.selenium.WebDriverException: Failed to execute command browser goto:",
         "Driver info: com.qmetry.qaf.automation.ui.webdriver.QAFExtendedWebDriver",
      ],
      
   },
   {  
      "CustomError" : "Just a Test".
      "StackTraceErrors":[  
         "Backend issue"      ]
      "Tags" : ["Backend issue"]
   }
] 

See also

• Adding a Failure Reason to the Test Report
  
• Quantum Release Notes 1.20