Optical and electronic designs for optical deep learning networks

Deep learning object detection has revolutionised AI in the past ten years. However, computer servers have huge, and growing, energy requirements. The best in class compute training requirements are currently doubling every 3.4 months. For object detection networks the vast bulk of the computation is taken up by convolutional layers. Since a lens can perform a two dimensional Fourier transform of the incoming optical wavefront, correlation and convolution operations have been successfully demonstrated in the past with a 4-f based optical system. These have the potential of a very large computational bandwidth. The power requirements are then from the coherent light source, spatial light modulators and sensors. This is typically dominated by the camera but still only about 5W. However, network designs such as ResNet, VGG and Inception do not map well onto a pure optical domain. Convolutional layers are not just a single operation but the sum of several convolutions, with an additional bias and a non-linear activation function. In this paper we develop a Python framework for simulating optical deep learning using Pytorch. This is used to develop and demonstrate optical designs that take a hybrid approach to the network by combining low-powered electronic signal processing with parallel optical Fourier transforms, the latter performing the majority of the calculations. We introduce a novel activation function and demonstrate that the performance remains intact. This allows for arbitrary network designs to be introduced. Coherent optical processing methods however suffer from various noise sources of which laser speckle is a major source. We model speckle effects in a simple four layer convolutional network which was able to achieve a 73% accuracy, but this drops off rapidly to 10% with a speckle contrast of 0.5. Various speckle reduction methods have been considered and shown to restore the accuracy.