I posted a LinkedIn note in October 2026 summarising the top 5 parameter‑efficient fine‑tuning (PEFT) techniques I use for large language models. The methods all rely on inserting low‑rank matrices A and B into each linear layer and varying which of them are trained, shared, or frozen to reduce compute and memory costs.
LoRA adds two low‑rank trainable matrices A and B alongside each original weight matrix W; even for the largest of LLMs, LoRA matrices take up a few MBs of memory.
LoRA‑FA (FA stands for Frozen‑A) freezes matrix A and only updates matrix B.
In VeRA, A and B are frozen, random, and shared across all layers.
Delta‑LoRA injects the difference (or delta) between successive A·B products into W.
Authors of LoRA+ found that setting a higher learning rate for matrix B results in better convergence.
These design choices trade expressive capacity for parameter efficiency: freezing A reduces activation‑memory traffic (LoRA‑FA), sharing random A and B across layers cuts trainable parameters dramatically (VeRA), and using asymmetric learning rates exploits the different optimisation roles of the two factors (LoRA+).
For more context on my broader LLM customization experiments see my LinkedIn LLM customization methods page.