LLM微调Fine-tuning完全指南

1
原理：在原始权重 W 旁边添加低秩分解 ΔW = BA
2
      训练时：只更新 A 和 B，不更新 W
3

4
W' = W + ΔW = W + BA

1
量化：模型权重从 FP16/BF16 → INT4
2
     训练时：反量化回 BF16 计算
3
     推理时：可直接用 INT4

1
Transformer Layer:
2
  Input → [原始注意力] → [Adapter] → Output
3
                 ↑ 冻结        ↑ 可训练

1
// 格式 1：Alpaca 格式
2
{
3
  "instruction": "将以下中文翻译成英文",
4
  "input": "今天天气真好",
5
  "output": "The weather is really nice today."
6
}
7

8
// 格式 2：ChatML 格式
9
{
10
  "messages": [
11
    {"role": "system", "content": "你是一个翻译助手"},
12
    {"role": "user", "content": "翻译：Hello"},
13
    {"role": "assistant", "content": "你好"}
14
  ]
15
}
16

17
// 格式 3：ShareGPT 格式
18
{
19
  "conversations": [
20
    {"from": "human", "value": "问题..."},
21
    {"from": "gpt", "value": "回答..."}
22
  ]
23
}

1
# 常见清洗步骤
2
1. 去除 HTML 标签和特殊字符
3
2. 去除过短或过长的样本
4
3. 去除重复内容
5
4. 去除个人隐私信息
6
5. 过滤低质量内容（可以用 Reward Model）

1
lora_config = {
2
    "r": 8,              # 秩，越大参数量越多
3
    "lora_alpha": 16,    # 缩放因子
4
    "lora_dropout": 0.05,
5
    "target_modules": [  # 目标模块
6
        "q_proj", "k_proj", "v_proj", "o_proj",
7
        "gate_proj", "up_proj", "down_proj"
8
    ]
9
}

1
from transformers import Trainer, TrainingArguments
2
from peft import LoraConfig, get_peft_model
3

4
# 配置 LoRA
5
lora_config = LoraConfig(
6
    r=8,
7
    lora_alpha=16,
8
    target_modules=["q_proj", "v_proj"],
9
    lora_dropout=0.05,
10
    task_type="CAUSAL_LM"
11
)
12

13
# 应用 LoRA
14
model = get_peft_model(base_model, lora_config)
15
model.print_trainable_parameters()
16
# trainable params: 8,388,608 || all params: 6,738,415,616 || trainable%: 0.124%
17

18
# 训练
19
trainer = Trainer(
20
    model=model,
21
    train_dataset=train_dataset,
22
    args=training_args,
23
)
24
trainer.train()

1
# 合并 LoRA 权重到基座
2
from peft import PeftModel
3
merged_model = model.merge_and_unload()
4

5
# 量化导出（可选）
6
quantized = merged_model.quantize(4)  # INT4
7

8
# 保存
9
merged_model.save_pretrained("output")