Skip to content
Snippets Groups Projects
Commit 5f27142f authored by Vlad-Andrei BĂDOIU (78692)'s avatar Vlad-Andrei BĂDOIU (78692)
Browse files

Merge branch 'refactor/structure' into 'main'

Add QLoRA paper, move LoRA paper to fine-tuning

See merge request !3
parents be6923fe bebbb65c
1 merge request!3Add QLoRA paper, move LoRA paper to fine-tuning
...@@ -12,12 +12,9 @@ ...@@ -12,12 +12,9 @@
oct. 2023)](https://arxiv.org/abs/2310.11453) oct. 2023)](https://arxiv.org/abs/2310.11453)
## Quantization ## Quantization
- ["LoRA: Low-Rank Adaptation of Large Language Models" (Hu et al. - jun.
2021)](https://arxiv.org/abs/2106.09685) - introduces fine-tuning using the
LoRA algorithm, which helps train models for downstream tasks faster
## Prompt engineering ## Prompt engineering
- ["Language models are few-shot learners" (Brown et al. (OpenAI) - may. - ["Language models are few-shot learners" (Brown et al. (OpenAI) - may.
2020)](https://arxiv.org/abs/2005.14165) - introduces the GPT-3 (175B 2020)](https://arxiv.org/abs/2005.14165) - introduces the GPT-3 (175B
parameters) model and the technique of prompt engineering by using few-shot parameters) model and the technique of prompt engineering by using few-shot
learning learning
...@@ -43,6 +40,12 @@ ...@@ -43,6 +40,12 @@
2023)](https://arxiv.org/abs/2307.09288) - LLaMA 2 2023)](https://arxiv.org/abs/2307.09288) - LLaMA 2
## Fine-tuning ## Fine-tuning
- ["LoRA: Low-Rank Adaptation of Large Language Models" (Hu et al. - jun.
2021)](https://arxiv.org/abs/2106.09685) - introduces fine-tuning using the
LoRA algorithm, which helps train models for downstream tasks faster
- ["QLoRA: Efficient Finetuning of Quantized LLMs" (Dettmers et al. - may
2023)](https://arxiv.org/abs/2305.14314) - builds on top of LoRA, and further
quantizes models to 4-bit, to reduce memory usage
## Benchmarks ## Benchmarks
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment