Learn about KZN Group’s involvement in building a medium scale IoT sensing platform, from the factory to the cloud, to detect faults in equipment.

Background

KZN Group was engaged to assist with the development of a productionised modular sensing platform with custom hardware consisting of a base unit that utilised LoRaWAN for connectivity and multiple pluggable sensors including accelerometers, temperature sensors and gas sensors. The data generated was used to remotely detect faults in equipment and monitor environmental conditions.

Works included the validation of hardware iterations; the building of factory test equipment to test manufactured devices on the production line; development of embedded firmware including device drivers, Digital Signal Processing (DSP) and network communications; development of Continuous Integration and Delivery of all components; and cloud telemetry and command and control of devices.

KZN Group worked alongside Electronics Engineers and technicians to deliver in an Agile fashion, delivering high value and robust solutions from the factory to the cloud.

Challenges

Moving from a Proof of Concept (PoC) to a medium scale productionised system has many challenges. Previous PoC iterations of the product resulted in a high failure rate after deployment, highlighting the need for hardware to be validated in the manufacturing process to eliminate defects, its provenance must be tracked from the factory to field deployment and rigorous test bench processes to stress test hardware and software iterations to simulate a device’s lifespan implemented.

A significant challenge in this engagement was operating within tight power, compute and memory constraints to deliver a product with a long life span with non-replaceable batteries, multiple modular sensors and measurements that will perform extensive processing.

Solution Overview

KZN Group delivered an end-to-end software solution using Agile methodology that:

• Took sensor measurements enabling equipment faults to be detected remotely,
• Utilised LoRaWAN networking to enable kilometer range communications,
• Validated hardware in the factory for quality assurance,
• Ingested telemetry and provided remote command and control of devices, and
• Enabled end-users to manage the device fleet through generated forms across a variety of configurations.

Design and Manufacturing

Pre-production Validation

In order to validate each iteration of the hardware KZN Group developed repeatable test procedures and test firmware. By utilising a vibration calibrator the accelerometer sensors were able to be characterised in terms of frequency response and the impact of resonance from their mechanical design understood and accommodated. A test bench was developed in order to simulate the expected lifespan of the devices to prove power consumption estimates were accurate and to expose any potential edge cases or infrequently occurring issues.

Additionally, during validation we leveraged our understanding of the hardware to troubleshoot and resolve communication issues over wired interfaces (SPI and I2C) between MCU, I/O expanders and sensor ICs.

Manufacturing Validation

During the manufacturing process we developed test procedures to be used in the factories that would allow defects to be detected and resolved early, speeding up delivery and increasing yield. With input from us a factory test jig was created with a small amount of custom hardware that would interface with the device under test that was driven by a small compute unit and a touchscreen. KZN Group wrote test software that was delivered to the equipment using Docker from Elastic Container Repository with authentication being provided by SSM Hybrid Agent to a compartmentalised AWS account to reduce possible blast radius. This software was used to imprint a read only device identity and credentials into the manufactured device whilst test results were recorded against its identity into a Manufacturing Inventory for provenance and future device provisioning. An IAM Authoriser on the API Gateway ensured only connections authenticated to the IAM Role assigned to the SSM Agent Hybrid Activation were authenticated and results were persisted in DynamoDB for later retrieval.

Edge Layer

To process data at the edge KZN Group developed a firmware application that ran on Mbed OS. This embedded software was capable of automatically detecting the connected sensors and took configurable measurements appropriate to the sensor type and use case. Once raw measurements were taken further processing was performed using Digital Signal Processing (DSP) in order to extract features, generating a reduced set of data that was able to be transmitted over the network whilst staying within the power consumption budget. The processing applied was configurable so that measurement data could be tailored to different use cases for particular devices.

There were a number of constraints that the design needed to accommodate:

• The hardware had non-replaceable or rechargeable batteries,
• The platform needed to maximise the lifespan of devices as physical access was limited,
• Handle a variety of measures from current sensors, as well as new sensor modules not yet developed,
• Some measures and processing required a significant amount of memory to perform.

Because of this significant effort was expended to optimise the firmware across storage, compute, memory and power consumption domains. This was achieved by splitting the firmware across multiple applications (production, testing and troubleshooting), recording power consumption differences across changes and profiling the resultant binaries statically and dynamically.

The software development activities were conducted using modern software development practices: all builds were automated, Continuous Integration and Delivery was performed and unit, integration and performance tests were implemented.

Fleet Provisioning

Out of the factory the devices have an identity and credentials, however they have not been personalised and enrolled into the tenant specific AWS account or LoRaWAN platform. To accomplish this a provisioning process was implemented - a QR code is laser engraved on to the device that contains it’s public identity and attributes. KZN Group developed a set of APIs that allowed for the Manufacturing Inventory to provide device details to the Device Management application, allowing for user interfaces to easily enroll the devices into the intended tenant platform.

Communication Layer

In order to achieve long range, low power communications the devices utilised LoRaWAN networking. KZN Group enabled the use of either the AU915 or AS923 band depending on the region deployed in, programmed the firmware to include an exponential back-off when connectivity could not be established to conserve battery life and configured the devices such that they can be used on any LoRaWAN platform that supports either of the available band plans. For enabling connectivity from the LoRaWAN platform to the target AWS account tenancies automation was created with CloudFormation to generate the required X.509 certificates. Additionally, since the devices were required to have additional Electromagnetic Compliance (EMC) certifications in some regions, KZN Group developed a test harness that allowed the certifier to exercise the device and confirm that it operated within standards.

Ingestion Layer

To allow downstream systems to integrate with the LoRaWAN devices an ingestion layer was developed to decode and encode the telemetry and control plane data. This application server utilised AWS Lambda and AWS IoT Core to abstract the network protocol concerns away from higher level downstream systems by providing an easy to consume JSON interface.

Analytics

In order to monitor device health diagnostics payloads were implemented that contained information such as sensor errors, transmit counts and cycles counts. By ingesting this information into CloudWatch these diagnostics were used to inspect individual device and fleet health allowing platform operators to maintain the fleet and ensure data quality expectations were met and that downstream users of the data could continue to receive the telemetry required for decision making.

Application Layer

Supporting users in configuring devices, for example to provide the parameters of accelerometer measures to match their use case, can become difficult when many device types, versions and attached sensor configurations need to be considered. To support this KZN Group developed an application to interface with the control plane of the devices - by abstracting the concepts of networking technology, device types, device versions and physical configurations it became simple to generate a schema describing the possible configurations available to a particular device at a point in time based on it’s last known state, allowing forms to be automatically displayed in any web or mobile client whilst retaining all domain specific logic in one centralised location.

This API was built using serverless technologies including AWS Lambda, API Gateway, AWS IoT Core and EventBridge. All builds and deployments were fully automated with CI/CD via GitHub Actions with code coverage being greater than 90%.

Future Enhancements

During development of the edge processing performed by the device new use cases were uncovered. One such use case was allowing Subject Matter Experts to view the raw waveform recorded by the accelerometers to enable them to apply their expertise to further identify equipment defects. Since the amount of data required to supply this is significant, the throughput of LoRaWAN networking is minimal and no reliability guarantees are provided special considerations must be made. A feasibility study was conducted and it was determined that it would be possible to emit these raw waveforms encoded using Forward Error Correction at a reduced frequency, allowing the SMEs to perform further analysis on detected faults as they arise.

KZN and IoT

KZN Group can help you deliver end-to-end IoT solutions using modern Agile development and AWS cloud best practices. We have real-world experience building bespoke IoT products and can assist you through hardware validation and process automation to firmware development and cloud analytics and applications. We can utilise our extensive experience with DevOps and Data and Analytics (including Data Lakes) to enable you to make the most of your data.

Contact us to learn more!