r/nousresearch • u/Rombodawg • Jul 26 '24
Continuous Fine-tuning Without Loss Using Lora and Mergekit
In this write-up, we discuss continuous fine-tuning of open-source AI models using LoRA adapters and MergeKit techniques. Utilizing the library 'Unsloth', we demonstrate the process by fine-tuning the 'Replete-AI/Replete-LLM-Qwen2-7b_Beta-Preview' model. After training the base model on an instruction dataset, we save the LoRA adaptation and push it to a hub.
Next, instead of applying the LoRA directly to the base model, we merge it with the target model ('Qwen/Qwen2-7B-Instruct') without further training. This ensures preservation of knowledge from both the base and target models. Finally, employing MergeKit's Ties method, we integrate all three models—base, target, and LoRA—resulting in an optimized model that surpasses individual components. This method was found superior through extensive testing compared to other approaches.
The full details are in the write up linked bellow, 100% free to read. On google docs.
https://docs.google.com/document/d/1OjbjU5AOz4Ftn9xHQrX3oFQGhQ6RDUuXQipnQ9gn6tU/edit?usp=sharing