The benefits of an automated approach to standards testing and certification

Data networks are now intrinsic to our everyday experiences. Automobiles are data centres on wheels with hundreds of sensors and control systems in constant communication. Smart appliances, lighting, climate control and security systems manage our home environments invisibly.

These experiences are so seamless that it is easy to forget how much work goes into making such a wide variety of devices from disparate manufacturers work together.

That interoperability is made possible by standards. They govern everything from how microprocessors interact with storage systems to the way cell phone towers hand off signals to roaming devices without disrupting conversations. There are more than a dozen in the field of wireless connected devices alone.

The challenges with traditional standards processes

In most cases, the development of standards is the domain of industry consortia composed of manufacturing representatives. They collectively define requirements and test solutions to ensure that products interoperate.

This process can be complex and prone to error considering how quickly technology changes. Bluetooth users may remember the many incompatibilities that made it difficult or impossible for devices to exchange data with each other in that standard’s early days. Lack of clarity or disciplined methodologies creates a risk that products will reach the marketplace without being fully tested.

Standards also evolve. Bluetooth, for example, has undergone numerous revisions over the years to account for lower power requirements, greater range and the need to support audio output, a popular use for which the original specification was never intended.

Compliance and testing are often decentralized processes. Standards creators are responsible for administering test suites and verifying compliance with standards that are maintained by a certification authority. Unfortunately, there have historically been structural limitations to the certification process that make this difficult.

For new standards, in particular, there is a chicken-and-egg problem. Without fully compliant devices on the market, testing can be a bit of a guessing game. Until such a “golden device” is available, manufacturers must estimate their level of compliance. Certifying organizations must search for solutions that allow the test tool to be adapted to the specifications with consistent quality and performance.

The result is that, paradoxically, an IT industry that thrives on automation has long used manually intensive processes to certify compliance with communication standards. Test suites may be hardcoded into test tools and require any changes in the standard or the tool to be made by hand. Changes to test scripts have long been documented on paper, creating the risk that human error could cause products to reach the market without having passed the full suite of authorized tests.

The solution to improve standards processes

Fortunately, automation is finally coming to the test and certification process. An automated test framework using scripts written in a flexible, interpreted language like Python enables test suites to be easily customized for iterative development.

For example, if a “golden device” is unavailable, software simulations can be used as a proxy. Test scripts are easily changed as standard criteria are finalized and vendors can work in simulated environment for precertification test runs during the development process.

Automation enables engineers to audit devise on a step-by-step basis, with the ability to pause a script and make changes before execution has been completed.

Once standards are in place, certification authorities can send test scripts to device makers that have been signed by a trusted authority to ensure authenticity. Any changes are automatically logged so that there is no possibility of intentional or inadvertent tampering.

The results

These advances yield dramatic improvements in productivity. One networking consortium reported that it went from dedicating four people to investigate a single test case over three days to verifying 25 test cases in less than five minutes.

For example, the FiRa Consortium adopted an automated approach to enable certification of the FiRa Ultra-Wideband (UWB) specification. The standard is considered a game-changer in the rapidly evolving IoT ecosystem because of its ability to determine the relative position of peer devices with a high degree of accuracy over distances of up to 200 meters with low power consumption and high reliability.

Working with a team of experienced Comarch engineers, the consortium quickly developed a monitor, control, test and tune solution consisting of hardware and software, as well as sample devices for testing. The software was built on top of the Comarch Automated Test Framework with dedicated firmware created to fulfil the standard’s requirements. Python test scripts were then developed for certification. A sequencer was deployed to allow flexible testing and the modular architecture allowed for the segments to be added easily, a key capability in a rapidly changing environment.

Manufacturers were able to easily modify or add test scripts for their own development purposes with the final set of input files protected by a certifying authority. The FiRa Consortium was thus able to provide certification tools in short period of time.

Delays in the standards certification process hurt everybody. Manufacturers are unable to bring products to market, and buyers are deprived of the choice, cost benefits and functional variety of a thriving market. Automation drastically improves productivity and reduces time-to-market, yielding dividends that benefit everyone.

Comarch has been a trusted partner in the certifications ecosystem since 2007. Discover how the company has helped redefine and enhance standards processes with its Automated Test Framework by visiting: https://www.comarch.com/professional-services/certification-alliances/