How does LoRA fine-tune a large model cheaply?
- Trains small low-rank adapters
- Retrains every weight in 8-bit
- Prunes heads after training
- Distills into a smaller model
LoRA freezes the base weights and learns a low-rank update B*A added to selected weight matrices.
Play AI & ML ▸