Let's take an example of a chatbot scenario and describe how to build a scalable and robust test suite to provide the needed coverage through test automation.
Here are the steps one would take to fulfill the test requirements:
- Create in your script a dictionary of user requests and response validations. For the most part, these would be a set of strings (see comment at the end)
- Use the Siri Inject, or text-to-speech function and inject the resulting audio file into the device
- For validation, there are a number of options, as typical interfaces would have both audio and textual (on-screen) response
- Conduct native object or visual validation on the response to validate correctness and responsiveness of the outcome
- Record the audio and use the speech-to-text to validate the vocal response
- Voice variations: every chatbot needs to consider accents, stuttering, "Um" and "ah" etc. For that purpose, one can record audio files and use audio inject in order to introduce those into the script
- Audio quality (for media applications): Another function provided is audio quality, where the audio is recorded and fed into an automated algorithm. Two options are offered, with or without reference. The former could be used to compare a pre recorded song (as example) vs. one that is being played live. A particularly interesting scenario to test is validate streaming quality in the presence of sub-optimal network conditions via Wind Tunnel Application Performance Testing. The latter (reference-free audio quality) can be applied to live streams where one cannot anticipate the song that will be played, as example.
- Voice call quality and Visual voicemail: create a phone call between two devices, inject audio and measure the quality on the other end. For visual voicemail, hang the call and validate the text that shows in the voicemail.
- IVR and 911 monitoring: create a phone call and interact with/validate the remote end via audio inject and audio record/speech-to-text