Demonstrators

SCD 6.2: Neuromorphic sensor fusion

IMEC

INNAT, TUDELFT

Modern sensor pre-processing and fusion methods rely on rising number of sensors to the sensor suite. This causes a downstream data deluge with diminishing returns in terms of aggregate perception quality. In this demonstrator we showcase an event-based neuromorphic sensor fusion architecture, which aims to solve this bottleneck by enhancing the data quality coming off the sensors. All the involved partners will work together on the design and implementation of energy-efficient neuromorphic accelerators for edge intelligence. The architectures will incorporate low-level neuromorphic intrinsic, e.g. complementing spiking neuron models by more detailed models of synapses and dendrites, and microarchitectures to support fast, efficient, and robust inference and learning capabilities at the very edge. Development efforts will span the flow from architecture exploration, down to system-level design and prototyping, as well as physical tape out designs. By doing so the benefits of spiking neural networks: better focus on relevant senor data with subsequent lower computational energy demand as well as the possibility to perform online learning to adapt to changing environments and changes in sensor behaviour shall be explored. We will demonstrate how event-based inference processing reduces energy consumption for a select number of use cases, using a single sensor and by fusing the data from different sensors. Different approaches will be shown, including digital techniques (where all weights in the network are stored as digital numbers and all computations are carried as logic operations) and analogue techniques (where analogue voltages carry the values). The single sensor solution will be sensor agnostic and fusion solution will demonstrate the constructive performance for increased reliability of the system.

Demonstrator SCD 6.2 consists of two main parts sharing common spiking neural network technologies. The first demonstrator (SCD 6.2.A) focuses on a HW/SW co-design based on a spiking neural network and embedded processing for automatic detection, tracking, and classification of vulnerable road users and vehicles. This subdemonstrator will utilise the developed and fabricated scalable neuromorphic accelerator hardware prototype for low-power, high performance deep edge processing for radar-based target classification and supportive SoC evaluation platform for the rest of the required processing. The second demonstrator (SCD 6.2.B) focuses on the development of a programmable analogue-mixed signal accelerator platform for always-on detection and recognition of time-series patterns in sensor data. The demonstrator utilises spiking neural network algorithms, implemented in an accelerator fabric for low-latency, ultra-low-power processing. The fabricated chip will utilise the SNN model for classification of time-series models was developed to run on the chip – uses audio as a representative case of time-series patterns.

SCD 6.2.A Neuromorphic accelerator hardware prototype

SCD 6.2.B Neuromorphic accelerator hardware prototype