High-performance real-world optical computing trained by in situ model-free optimization

Guangyuan Zhao1, †, , Xin Shu1,†, Renjie Zhou1

1The Chinese University of Hong Kong   †Equal Contribution   Corresponding Author

ICCP&TPAMI 2024 Best Paper Award of ICCP'24

teaser
Left: We treats an optical computing system as a black box and back-propagates the loss directly to the optical computing weights' probability distributions, circumventing the need for a computationally heavy and biased system simulation. Right: Visualization of experimental input and output of the optical computing system.

Abstract

Optical computing systems provide high-speed and low-energy data processing but face deficiencies in computationally demanding training and simulation-to-reality gaps. We propose a gradient-based model-free optimization (G-MFO) method based on a Monte Carlo gradient estimation algorithm for computationally efficient in situ training of optical computing systems. This approach treats an optical computing system as a black box and back-propagates the loss directly to the optical computing weights' probability distributions, circumventing the need for a computationally heavy and biased system simulation. Our experiments on diffractive optical computing systems show that G-MFO outperforms hybrid training on the MNIST and FMNIST datasets. Furthermore, we demonstrate image-free and high-speed classification of cells from their marker-free phase maps. Our method's model-free and high-performance nature, combined with its low demand for computational resources, paves the way for accelerating the transition of optical computing from laboratory demonstrations to practical, real-world applications.

Comparison of training strategies

figure_description
Comparison of strategies on training optical computing systems along the axes of in situ training capability (in situ), in silico computation overhead (computation overhead), the requirement on the recording of the input object to the task (input image free), required in silico training time, the dimensionality of trainable parameters (dimensionality), requirements on intermediate measurement and whether or not use gradient-based update that is more efficient (gradient). Simulator-based training (SBT), Hybrid model-based training (H-MBO), Learned model-based training (L-MBO), Intermediate measurement optical backpropagation (IMOB), Forward-forward training (FFT), Trial and error, Genetic algorithm (GA), Gradient-based model-free optimization (G-MFO).

Results

figure_description
figure_description

Video

Citation

@article{Zhao2024MFO,
	 title={High-performance real-world optical computing trained by in situ gradient-based model-free optimization},
	 author={Zhao, Guangyuan and Shu, Xin and Zhou, Renjie},
	 journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
	 year={2024},
	 publisher={IEEE}
}