Photos and videos are usually produced by capturing photons with digital sensors. Digital cameras consist of millions of pixels that form images by detecting the intensity and colour of the light at every point of space. 3D images can then be generated either by positioning two or more cameras around the subject to photograph it from multiple angles, or by using streams of photons to scan the scene and reconstruct it in three dimensions. Either way, an image is only built if one gathers spatial information of the scene. Reserchers have found another way using a simple one-dimesional ToF sensor and a neural network AI.
In a paper published in the journal Optica, researchers based in the UK, Italy and the Netherlands describe how they can make animated 3D images by capturing temporal information about photons instead of their spatial coordinates. Their process begins with a simple, inexpensive single-point ToF detector that measures the time photons produced by split-second flash of a pulse of laser light take to bounce off each object in any given scene and reach the sensor. The time taken for each reflected photon to reach the sensor is proportional to the distance between the sensor and object.
The information about the timings of each photon reflected in the scene (or temporal data) is collected in a very simple graph.
These graphs are then turned into a 3D image with the help of a sophisticated neural network algorithm. The researchers ‘trained’ the algorithm by showing it thousands of different conventional photos of the team moving and carrying objects around the lab, alongside temporal data captured by the single-point detector at the same time.
The neural network ‘learns’ how the temporal data corresponds to the photos and is then capable of creating highly accurate images from the temporal data alone. In the proof-of-principle experiments, the team managed to construct moving images at about 10 frames per second from the temporal data, although the hardware and algorithm used has the potential to produce thousands of images per second.
