Testing Alexa (Amazon Echo) weather application

Question

You’re a test engineer at Amazon working with the team who creates skills on Alexa (Amazon Echo). Your team has created a “weather” skill where you can ask Alexa about the weather and it responds with relevant information.

How would you test this skill? What would your test cases be?

Answer

Test automation is possible here and efficient if we automate on the data layer. Before you start answering the questions, you should ask questions to limit your scope of the problem domain. See some probing questions below.* Automation? Test it on the data layer

Probing questions

  • What are the possible wake words for the application.
  • In which countries is the application going to be available.
  • On what devices and their versions is this application available for?
  • What are the default weather attributes(temp, humidity, wind etc) that devices is going to speak.

  • Draw the rough architecture of the application to find out the different points of data interchange and discuss with interviewer. See, for example in the below image:

Alexa Weather App - Rough Architecture

  • A user speaks to Echo with voice asking for the weather.
  • The voice is sent to the Alexa platform and converted to text and sent to AWS Lambda (serverless) for execution.
  • While executing on Lambda it talks to some other service (say a weather service) to retrieve weather details.
  • The weather details are converted back to audio and sent to the user.

Test Cases

  • Test in different languages that application supports.
  • Examples of possible questions to ask to the voice application -

    • Alexa! Tell me about the weather
    • Alexa! Weather
    • Alexa! How is the weather?
    • Alexa! Weather in abc city.
    • Alexa! Weather in abc country.
    • Alexa! Resume my weather book. (negative testing)
    • Alexa! Weather music (negative testing)
    • Alexa! Play weather (negative testing)
  • If we know the location scope of the weather forecast, then on the basis of requirement possible test cases can be:

    • Test “Alexa! Weather in abc city, abc state, abc country”.
    • Test for cities across different countries.
    • Test for different states across different countries.
    • Test for “Alexa! Weather at my location?” (change current location few times and test)
    • Test for weather at a specific location within the city.
  • Test for weather in the city whose names are changed in the constitution recently.

  • Test for cities who have same name.

  • Test for any remote city weather.

  • Test using remote city as my current location and ask “Alexa! Weather at my location? “

  • Test for a city that doesn’t exist.

  • Ask the weather application for weather on day basis

    • “Alexa! weather for today”
    • “Alexa! weather for tomorrow”
    • “Alexa! weather for entire week”
    • “Alexa! weather for month” (negative testing)
    • “Alexa! weather for next few days”
    • “Alexa! how was the weather yesterday?” (negative testing)
  • Ask the weather application for weather on time basis

    • “Alexa! weather at x time today”
    • “Alexa! weather after 2 hrs from now”
  • Test Alexa with diff name (Amazon, Echo etc)

    • When Echo settings has these name enabled test whether the skill invoked or not.
  • Test maximum number of users of same location using the weather application (performance at peak traffic)

  • Test above maximum number of users allowed using the weather application.

  • Test maximum number of users from different location using the skill (performance at peak traffic)

  • Test the overall skill execution time is within the allowable range (on the Lambda side).

  • Test the response from the weather application when the server or or any dependency is down.

  • Test/Monitor the execution time and memory footprint on AWS Lambda if the weather application is serverless.

  • Test the response from the application when the dependencies have partial data that is needed as mandatory attributes for application to speak.

  • Test the applications response time in different internet connection bandwidth

  • Test the application for telling the temperature in different units (fahrenheit or celsius).

  • Test the application for indirect weather question for eg. “Alexa! should I take an umbrella out”.

  • Test whether the command “Alexa! stop” works while she speaks about the weather.

  • Test whether the application supports globalization for eg. application states the temperature in different units on the basis of which country it is located at.

  • Test whether the application supports localization for eg. it speaks in different language on the basis of which country it is located at.

  • Test the application by asking the question with varying speed(fast, slow or with pauses).

  • Test the application using short form of the state, city or country.

  • Test when the application does not respond, application’s team is notified about it with relevant details.

  • Test the applications response when the server stops responding.

  • Test the reliability of the application.

  • Test the accuracy of the application for eg. the data that is spoken by application is same as the data present in the database that the skill fetches from.

  • Test the application to make sure that there are no conflicting app , the one same/similar wake words.

  • Test whether the application is able to handle fuzzy data. for eg. dependencies give you input data, fake the data values to its min or max value and check what is the output.

  • Test weather for alien places “Alexa! how is the weather on mars?”


See also