Every day, millions of customers use devices to watch Prime Video in their living rooms. These devices include smart TVs, consoles, set-top boxes, or USB streaming sticks, and have grown significantly in number and popularity over the past few years, with hundreds of new devices being launched annually. The landscape of living room devices is characterized by fragmentation, with relatively few standards. Device manufacturers also often install proprietary operating systems, communication protocols, and interfaces.
With such a large and diverse mix of devices, how can Prime Video efficiently test them all and ensure that our customers always get the best experience from our application?
Testing has long been a challenge for both application developers and device manufacturers (often resolved by countless hours of manual tests) because currently available solutions for test automation are neither reliable nor consistent. This prompted Prime Video, as part of an open-source group (see footnote for more information), to create the Device Automation Bus (DAB) protocol that allows developers to programmatically interact with any living room device.
DAB is a simple protocol that defines a standard way to connect and send commands to a device, and receive a response. For example, DAB can be used to send a command to start the Prime Video application and receive a success or failure message in response. A device with DAB capabilities allows application developers and device manufacturers to automate the testing of applications on the device, reducing time and effort, and thereby improving quality. At Prime Video, we use DAB to automate the testing of our application on new devices before they reach the market.
Testing living room devices is challenging
User interface (UI) testing automation for web and mobile applications is made easy by the rich choice of open-source test automation tools available, such as Selenium (web) or Appium (mobile). With these tools, developers can write tests in a matter of minutes and run them on any browser or mobile device. The same test can be written once and run on multiple devices or browsers, with no changes.
Unfortunately, this isn’t the case for living room devices. Currently available tools lack both reliability and interoperability. For example, IR blasters (devices that send commands by infrared code just like a remote) are a common way to automate testing, but they are not reliable and don’t provide any feedback mechanism for the developer to know if a command has been run or not.
Manufacturers typically provide some tools to interact with their devices (for example, Roku drivers or Android Debug Bridge), but they are not compatible with each other. To test an application across multiple devices, developers have to write their own device abstractions, which is time-consuming and difficult to scale with an ever-increasing number of devices. Moreover, the set of operations that can be automated with these tools are not standardized and can vary significantly from device to device. As a result, developers have to write and maintain device-specific tests.
This problem affects not only application developers but also device manufacturers, who want to test new devices and firmware updates with all the applications installed on them. Without testing automation tools that work across applications, they have to physically ship devices to each application developer for testing or manually test each application themselves. These are extremely costly and time-consuming solutions. Manual testing is also prone to error and the quality of testing cannot be guaranteed.
Device Automation Bus protocol
Device Automation Bus (DAB) is a new open-source protocol that is the result of collaboration, with the goal of making test automation on living room devices easier. Version 1.0 of the DAB protocol specification was released in March, 2021.
DAB allows application developers and device manufacturers to programmatically interact with any remote device that supports the protocol. Supported operations include starting and stopping applications, sending input key presses, retrieving device information, and capturing metrics such as CPU and memory usage.
The DAB protocol is built on top of the publish-subscribe network protocol – Message Queue Telemetry Transport (MQTT). MQTT is the standard for device communication in many application domains, such as automotive, smart homes, and consumer electronics. MQTT clients publish messages to “topics” via a message broker. The message broker delivers the messages to all topic subscribers. The following diagram shows this workflow.
DAB defines the list of topics and the expected JSON payloads for request and response messages. For example, to send a left-button key press, the test automation client will publish a message to the dab/input/key-press
The subscribed device receives the message, runs the command, and publishes a response topic message, including the status code. This feedback loop improves test reliability because developers can determine if and when commands are successfully run on the device, and if a test failed because of an issue with the device rather than the test itself.
The manufacturer effort required to implement the DAB protocol is small and provides more rigorous automated testing and no manual intervention. The end customer will benefit from quality improvement, while the manufacturer benefits from reduction of testing costs and faster time to market.
With DAB, both application developers and device manufacturers can write device-agnostic test scripts that run on any DAB-supported device without customization or duplication. They can capture device metrics, and they can test user journeys that previously would have been very difficult to automate, such as switching between applications and changing device settings.
How we use DAB at Prime Video
At Prime Video, we want to ensure that customers have the best experience using our application, regardless of the device it is running on. To achieve this, we partner with device manufacturers and require each device model to undergo a rigorous certification process. This involves a special test application and test suite that verifies that the device meets Prime Video’s functional and performance bar. Certification tests are run on devices that are physically located in our device partners’ labs. Testing is orchestrated by our backend test automation software.
When a device partner wants to run a test run on one of their devices, they initiate it via our web portal using a device ID. This launches our test application which connects to our services, sends commands to the application and controls the test execution. Before DAB, we could only control the test execution inside our application and we needed an engineer to manually perform all the operations on the device, including starting and stopping the application. Now, we can interact with the device via DAB, negating manual intervention and streamlining the process.
Because the network interfaces of devices in partner labs are not publicly exposed to the internet, we have developed a DAB Harness utility that can be downloaded and run in the same local network as the device. This pairs with a specific device and connects to our service, forwarding DAB commands between the service and the device.
What’s next?
DAB represents a big step forward in test automation for living room devices. By choosing to implement DAB, device manufacturers can significantly reduce the effort and time required to certify their new devices and new firmware releases, improving quality at the same time. Application developers can write tests without needing to know or implement device-specific protocols. They can leverage DAB feedback mechanisms to make testing more reliable and easier to debug.
DAB is an open-source protocol to which anyone can contribute enhancements or new features. DAB version 1.0 contains features that cover the basic automated testing requirements. However, future versions might include support for advanced features such as voice commands or pointer controls, the ability to capture screenshots or video, or to change device settings (for example, display resolution and audio output).
We expect that the DAB protocol will continue to evolve to reflect the constantly increasing capabilities of living room devices, and to increase the coverage of test cases.
Stay tuned for more from us!
[1] The DAB protocol was developed as a collaborative effort between Amazon, Netflix, and YouTube. You can find more information about DAB on the GetDAB website.