Researchers at the Vienna University of Technology have developed an image sensor with an integrated artificial neural network (ANN) capable of learning and classifying images within nanoseconds. The chip is a thousand times faster and a lot less power than conventional vision technologies. The sensor’s fast capture and processing of images does not consume power, since the photons themselves provide the energy for the electric current.
The researchers, supported by the European research project, the Graphene Flagship, devised sensors containing nine pixels, or the ‘neurons’, placed in a 3×3 array. Every pixel in turn consists of three photodiodes that provide three outputs. Each photodiode links its pixel to the other eight pixels. The current from each photodiode is determined by the intensity of incoming light and the voltage across it. Each neuron sums the individual currents coming from the other eight neurons, with the combined values then fed into a computer.
The device can classify images after a series of training processes, but it can also recognise a characteristic component or structure of an image from input data, without extra information.
The speed sets this device apart from conventional machine vision. Conventional technology is usually capable of processing up to 100 frames per second, with some faster systems capable of working up to 1,000 frames per second. In comparison, this system works with an equivalent of 20 million frames per second.
In combination with other technologies, the device can find many uses, including in fluid dynamics, high-energy physics, combustion processes or mechanical breakdown.
“We are considering other ideas: like improving the light absorption or extending the spectral range into the infrared. In principle, the capabilities of this device are not only limited to visual data – any kind of data could be (pre)processed with an artificial neural network in the sensor itself. For example, audio or olfactory neuromorphic sensors could be developed for rapid on-chip processing,” said Lukas Mennel, author of the study.