What is LoRA (Low Rank Adaptation Model) technique?
The LoRA method has been proposed by a team of Microsoft. This method focuses on fine tuning the Large Language Models only. LoRA is based on the principle of low-rank approximation, a mathematical model designed to simplify the intricacy and dimensions of expansive matrices by identifying smaller matrices. LoRA freezes the initial weights within the pre-trained LLM and then reduces the number of trainable parameters by learning pairs of these smaller matrices. This helps reduce the storage requirement for large language models adapted to specific NLP tasks without compromising the inference capability.
What are the advantages of the LoRA method in fine tuning LLM?
- Focuses on reducing the number of trainable parameters.
- Due to the...
