How to Build Winning Smart TV Apps With a Comprehensive Testing Strategy?

The pattern of content consumption has changed markedly in the last two decade. From mobile phones in the 90’s to smart phones to smart TVs, the demand for smart connected devices continues to grow. According to a Gartner consumer survey, worldwide, by 2018, 87 percent of the TVs shipped annually will be smart TVs. The figure below indicates the steep rise in the demand for smart TV from 2011 to 2017.

Source: Frost & Sullivan

Brands are foraying into Smart TV applications 

More and more brands are foraying into the smart TV app market. Entertainment and games are the biggest chunks of apps on Smart TV app stores, followed by social network and shopping apps. Apple TV is scaling up big time, and Android TV has seen about 5-10 million app installs to this day.

Great user experience is the definitive factor for the success of TV apps. Faulty apps with errors and bugs can play havoc with a brand’s reputation. Therefore, it is important that app developers conduct rigorous testing before pushing their apps on smart TV app stores, as the cost associated with testing is way less than the cost accrued in finding and fixing bugs and errors post production.

Achieving Quality in Smart TV Application Development

Testing Smart TV apps is a different ball game altogether when compared to testing mobile and tablet apps. Therefore a significant shift in testing approach needs to be made for testing and QA of smart TV apps.

User perspective is an important consideration for developing smart TV apps. Hence interface testing with people is important. While automated testing and emulators have become a norm nowadays given the amount of time it saves for testing the varied use cases, manual testing should be complemented to consider factors that automated test environments cannot. The size of the screen, distance between the user and the screen are important considerations while testing the visual rendering, image size, and placement of elements on the screen. Emulators are not as effective when testing design and layout. Therefore it is always better to use real TV. It is important to test out the UI in multiple settings.

What makes Smart TV apps testing more complex is the availability of a sheer number of smart TV vendors, numerous versions, and models, the number of testing platforms and Operating Systems. It is essential for app providers to get their application tested for the particular Smart TV model, the platform used and the operating system/ browser that it supports.

Typical Test Types

• Regression testing: It is important for detecting errors or bugs in functional or non-functional areas that might arise after code modification
• Functional testing:  It is a QA process and a type of black-box testing. It tests the specifications of software components and typically focuses if the output is as per requirement
• Smoke testing: Although such a testing is not very in-depth, it is preferable to pass all QA builds through smoke testing to weed out basic errors
• Browser compatibility testing: App is thoroughly checked under different versions of browsers
• Performance testing of API’s: API integration testing and performance testing can be done using JMeter
• Usability Testing: Usability can make or break the user experience. With usability testing. the application’s flow and its navigation is checked, and UX is improved
• Crowd Testing: Multiple users test the app for wider inputs on errors

Considerations for Smart TV Application Testing

• Abide by Platform Store Guidelines– All Platform Stores have guidelines that applications need to follow before they can be listed on the App Store. It is of primary importance for testers and developers to abide by these guidelines to avoid application rejection. Most companies have basic guidelines such as having instant and distinct control feedback, not keeping unnecessary screen levels, keeping consistency in navigation, screen layouts, controls and many others. Here are guidelines for Roku, Apple TV, Tizen, and Android TV 
• Maintain bugs in a tracking tool – It is important to keep track of all issues and maintain them at a centralized database or use a bug tracking and management tool. It will help to ensure that the bugs are fixed timely 
• Complement Emulators with manual testing– Emulators come in handy for smart TV app testing as it is not possible to test on all versions of smart TVs as they are pretty expensive. However, some platforms like ROKU do not provide emulators and some existing emulators of platforms does not support testing of all necessary functionalities such as video streaming or network checking 
• Network use cases– Sometimes testers might overlook scenarios where network connectivity is minimal. Under such circumstances, the app might freeze, close abruptly or display other irregularities. Testers need to test these use cases so that the right error messages are displayed in case of low network connectivity 
• Resolution issues– Smart TV apps are built using multiple technologies. During Testing, it might so happen that an app may work fine on one smart TV with a particular resolution and freeze on another as smart TVs do not support pixel CSS 
• UI issues– It is much more than cosmetic fixing. For smart TV apps, UI testing is essential. While testing on different platforms, design issues are bound to crop up. Smart TV apps should have minimalistic feature sets with minimal user steps 
• Usability– Testers need to look into the seamlessness of the app experience from the end-user perspective. Developers and testers should focus on keeping the app experience minimalistic with single click functionalities, which means a minimal number of buttons on the remote. Navigation should be simple and easily comprehensible, visual rendering should be of high quality, sharp (alignment and spacing should be taken into consideration) and should adjust as per the screen size available and resolution should be optimal. Loading and buffering should be indicated by the loading indicator and the search capability should allow enriched browsing and content discovery.

– Navigation movements should be predictable

– Limit screen transitions

– App should not require a separate guide

– Keep the focus clear including user’s existing on-screen and selected component

Case Highlight:  Smart TV Application Testing for Jadoo TV                                    

TO THE NEW has expertise in Smart TV application development for Android TV, Apple TV, Amazon Fire, and Roku TV as well as on Samsung Native, Samsung Tizen platforms. Recently, we developed Jadoo TV’s smart tv applications for 7 different platforms.

Jadoo TV delivers Live and On-Demand content to viewers via its proprietary Internet based set-top box (STB). The client wanted to take its services to Smart TV viewers for higher engagement.

We custom developed a highly engaging and intuitive app for seven platforms including Tizen, Samsung Orsay, LG, Roku TV, Android TV, Amazon Fire and Apple TV. We used platform specific languages such as Java, HTML, Brightscript, and Swift followed by rigorous testing before going live with these applications on respective stores.

Over 500 test cases were created, and rigorous ongoing testing of applications was done before pushing it to the app store to reduce commercial and technical risk-

• Each platform-specific guidelines were met. Developer, store and QA checklist were checked thoroughly before submission in store
• Exhaustive test cases developed for all platforms and complexities were defined. Installation and uninstallation of app, splash validation, TV form factor support, UI design, navigation, testing of menu buttons, focus and remote functionality to validate gesture were carried out in detail
• Performance testing was done under differing network scenarios; load testing and API integration testing was conducted after each build. Smart TV apps do not come with dedicated CPU and RAM for processing. Hence it is important to check the app responsiveness and put in place load indicator in case it takes more than 3 seconds to respond
• The team focussed on tests that are easily overlooked. They made sure there was no conflict in focus and navigation, undertook memory optimization, thoroughly checked remote functionality and keyboard handling, ensured smooth scrolling, validated localization (for Roku app) to see if channel is restricted to a particular region, checked in-app video streaming, privacy functionality for user opt-in, validate load time and other such critical functionalities

Takeaway

Smart TV app market is growing rapidly. Apple, Android, Roku, Tizen, LG, Opera and others are coming up with new features every passing day and their quality checklist is getting rigorous. Brands looking to enter the smart TV app market needs to focus on delivering a quality app to the app stores. Hence, developing multiple test case scenarios are of primary importance. Monkey testing does not investigate all test case scenarios as only certain test cases are picked up randomly by the testing team. If even a small bug is missed, it can compromise the end-user experience. Some of the companies offering Smart TV application development service also invest in Smart TV lab so that the testing is seamless and all the bugs are fixed across platforms and TVs. As there is a lot of effort in developing Smart TV app, it is imperative to go through platform guidelines thoroughly for avoiding future rejections and rework.

If you have some more insights and interesting use cases on smart tv application testing, please feel to share them with us.