$ ./guess_the_languageAI & ML
AI & ML · hard · #48

How does LoRA fine-tune a large model cheaply?

  1. Trains small low-rank adapters
  2. Retrains every weight in 8-bit
  3. Prunes heads after training
  4. Distills into a smaller model
LoRA freezes the base weights and learns a low-rank update B*A added to selected weight matrices.
Play AI & ML ▸