LoRA

LoRA, which stands for Low-Rank Adaptation, is a technique that allows efficiently adapting large pre-trained generative AI models to new domains or tasks. Rather than doing full fine-tuning of the model by updating all of the weight parameters on new data, LoRA works by learning only a low-rank update to the weights. This means tuning just a small subset of parameters compared to the full model size.

The low-rank update that LoRA learns is able to capture the crucial new adaptations required for the model to handle the specialized domain or task. For example, in natural language processing, LoRA has been shown to successfully adapt models like GPT-3 to new text styles, topics, and languages after updating only a hundredth or thousandth of the number of parameters as would be needed for full fine-tuning.

Similarly, for image generation, LoRA can adapt models like DALL-E 2 to generate new artistic styles, objects, and contexts by learning targeted low-rank weight updates rather than requiring full datasets and compute for fine-tuning. The key advantage of LoRA is that it enables efficient on-device adaptation on lower compute platforms like phones where full fine-tuning would be infeasible. Overall, LoRA provides a way to leverage the capabilities of large pre-trained generative models while specializing them for new personalized applications on demand through limited low-rank adaptation.