Symbolic Discovery of Optimization Algorithms
Discovers a simple and effective optimization algorithm, Lion, which is more memory-efficient than Adam.
Lion can improve the accuracy of various deep neural network models and save significant pre-training compute. It requires a smaller learning rate than Adam and is particularly effective for large batch sizes.
Universal Guidance for Diffusion Models
Proposes a universal guidance algorithm that enables diffusion models to be controlled by arbitrary guidance modalities without the need to retrain any use-specific components.
The proposed algorithm allows for diffusion models to be controlled by different guidance modalities such as segmentation maps, face recognition, object detection, and classifier signals. It eliminates the need to retrain use-specific components for each modality, making the process more efficient.