ml-unsloth-finetuning

Fine-tuning LLMs (Llama, Mistral, Qwen) efficiently

Unsloth Fine-Tuning

Scope: Fast LLM fine-tuning with Unsloth, memory optimization, multi-GPU, Flash Attention Lines: ~350 Last Updated: 2025-10-18

When to Use This Skill

Activate this skill when:

• Fine-tuning LLMs (Llama, Mistral, Qwen) efficiently
• Working in memory-constrained GPU environments
• Needing 2-5x faster training than standard methods
• Optimizing training costs with reduced memory usage
• Requiring rapid iteration on fine-tuning experiments
• Training with consumer GPUs (24GB VRAM or less)

Core Concepts

Unsloth Architecture

Speed Optimizations:

• Flash Attention 2 integration for 2x faster attention
• Custom CUDA kernels for backward pass optimization
• Optimized RoPE (Rotary Position Embeddings) computation
• Fused layernorm and cross-entropy operations
• 4-bit quantization with minimal accuracy loss

Memory Efficiency:

• Reduces VRAM usage by 30-60% vs standard training
• Enables larger batch sizes on same hardware
• Supports gradient checkpointing out of the box
• Efficient mixed precision (bfloat16) handling

Compatibility:

• Works with PEFT/LoRA for parameter-efficient training
• Supports Hugging Face Transformers ecosystem
• Compatible with Modal.com GPU workloads
• Drop-in replacement for standard fine-tuning

Supported Models

Llama Family:

• Llama 3.2 (1B, 3B), Llama 3.1 (8B, 70B, 405B)
• Llama 3 (8B, 70B), Llama 2 (7B, 13B, 70B)

Mistral Family:

• Mistral 7B v0.3, Mixtral 8x7B, 8x22B
• Mistral Small, Mistral NeMo

Other Models:

• Qwen 2.5 (0.5B - 72B), Qwen 2 (0.5B - 72B)
• Gemma 2 (2B, 9B, 27B), Phi-3.5 (mini)
• TinyLlama, OpenELM variants

Quantization Strategies

4-bit QLoRA (Recommended):

• Uses NF4 (Normal Float 4) quantization
• Maintains 99%+ of full precision performance
• Reduces memory by 4x vs FP16
• Allows fine-tuning 70B models on 48GB GPU

16-bit Training:

• Full bfloat16 precision for maximum quality
• Requires 2-4x more VRAM
• Recommended for models <13B with sufficient VRAM

---

Patterns

Basic Llama 3 Fine-Tuning

from unsloth import FastLanguageModel
import torch

# Load model with 4-bit quantization
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/llama-3-8b-bnb-4bit",
    max_seq_length=2048,
    dtype=None,  # Auto-detect (bfloat16 for Ampere+)
    load_in_4bit=True,
)

# Configure LoRA
model = FastLanguageModel.get_peft_model(
    model,
    r=16,  # LoRA rank
    target_modules=[
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj"
    ],
    lora_alpha=16,
    lora_dropout=0,  # Optimized for 0 dropout
    bias="none",
    use_gradient_checkpointing="unsloth",  # Unsloth's optimized checkpointing
    random_state=3407,
)

Dataset Preparation

from datasets import load_dataset

# Load dataset (Alpaca format)
dataset = load_dataset("yahma/alpaca-cleaned", split="train")

# Format for instruction tuning
alpaca_prompt = """Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
{}

### Response:
{}"""

def format_prompts(examples):
    instructions = examples["instruction"]
    outputs = examples["output"]
    texts = []

    for instruction, output in zip(instructions, outputs):
        text = alpaca_prompt.format(instruction, output) + tokenizer.eos_token
        texts.append(text)

    return {"text": texts}

# Apply formatting
dataset = dataset.map(format_prompts, batched=True)

Training with SFTTrainer

from trl import SFTTrainer
from transformers import TrainingArguments

trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=dataset,
    dataset_text_field="text",
    max_seq_length=2048,
    dataset_num_proc=2,
    packing=False,  # Can make training 5x faster for short sequences
    args=TrainingArguments(
        per_device_train_batch_size=2,
        gradient_accumulation_steps=4,  # Effective batch size = 8
        warmup_steps=5,
        max_steps=60,  # Or num_train_epochs=1
        learning_rate=2e-4,
        fp16=not torch.cuda.is_bf16_supported(),
        bf16=torch.cuda.is_bf16_supported(),
        logging_steps=1,
        optim="adamw_8bit",  # Reduced memory optimizer
        weight_decay=0.01,
        lr_scheduler_type="linear",
        seed=3407,
        output_dir="outputs",
    ),
)

# Train
trainer_stats = trainer.train()

Multi-GPU Training

# Use with torchrun or Modal multi-GPU
import modal

app = modal.App("unsloth-multi-gpu")

image = (
    modal.Image.debian_slim()
    .pip_install("unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git")
    .pip_install("xformers", "trl", "peft", "accelerate")
)

@app.function(
    gpu="a100:2",  # 2x A100
    image=image,
    timeout=7200
)
def multi_gpu_training():
    from unsloth import FastLanguageModel
    from accelerate import Accelerator

    accelerator = Accelerator()

    # Model automatically distributed across GPUs
    model, tokenizer = FastLanguageModel.from_pretrained(
        model_name="unsloth/llama-3-70b-bnb-4bit",
        max_seq_length=4096,
        dtype=None,
        load_in_4bit=True,
        device_map="auto"  # Auto-distribute across GPUs
    )

    model = FastLanguageModel.get_peft_model(
        model,
        r=32,  # Higher rank for 70B model
        target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
        lora_alpha=32,
    )

    # Train with accelerate
    # (SFTTrainer handles multi-GPU automatically)

Saving and Loading

# Save LoRA adapter only (small, ~100MB)
model.save_pretrained("lora_model")
tokenizer.save_pretrained("lora_model")

# Save merged model (full weights)
model.save_pretrained_merged(
    "merged_model",
    tokenizer,
    save_method="merged_16bit"  # or "lora", "merged_4bit"
)

# Push to Hugging Face Hub
model.push_to_hub_merged(
    "username/model-name",
    tokenizer,
    save_method="merged_16bit",
    token="hf_..."
)

# Load for inference
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="lora_model",
    max_seq_length=2048,
    dtype=None,
    load_in_4bit=True,
)
FastLanguageModel.for_inference(model)  # Enable fast inference mode

Inference After Training

# Enable fast inference (disables dropout, etc.)
FastLanguageModel.for_inference(model)

# Generate
inputs = tokenizer(
    [alpaca_prompt.format(
        "What is the capital of France?",
        ""  # Leave output empty for generation
    )],
    return_tensors="pt"
).to("cuda")

outputs = model.generate(
    **inputs,
    max_new_tokens=128,
    use_cache=True,
    temperature=1.5,
    min_p=0.1
)

print(tokenizer.batch_decode(outputs))

Modal.com Deployment

import modal

app = modal.App("unsloth-finetune")

image = (
    modal.Image.debian_slim()
    .pip_install(
        "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git",
        "xformers",
        "trl",
        "peft"
    )
)

volume = modal.Volume.from_name("model-volume", create_if_missing=True)

@app.function(
    gpu="l40s",  # Recommended: great price/performance
    image=image,
    volumes={"/models": volume},
    timeout=3600
)
def finetune_llama(
    dataset_name: str,
    output_path: str = "/models/finetuned"
):
    from unsloth import FastLanguageModel
    from trl import SFTTrainer
    from transformers import TrainingArguments
    from datasets import load_dataset

    # Load model
    model, tokenizer = FastLanguageModel.from_pretrained(
        model_name="unsloth/llama-3-8b-bnb-4bit",
        max_seq_length=2048,
        dtype=None,
        load_in_4bit=True,
    )

    # Apply LoRA
    model = FastLanguageModel.get_peft_model(
        model,
        r=16,
        target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
        lora_alpha=16,
    )

    # Load dataset
    dataset = load_dataset(dataset_name, split="train")

    # Train
    trainer = SFTTrainer(
        model=model,
        tokenizer=tokenizer,
        train_dataset=dataset,
        dataset_text_field="text",
        max_seq_length=2048,
        args=TrainingArguments(
            per_device_train_batch_size=2,
            gradient_accumulation_steps=4,
            warmup_steps=5,
            max_steps=100,
            learning_rate=2e-4,
            fp16=not torch.cuda.is_bf16_supported(),
            bf16=torch.cuda.is_bf16_supported(),
            logging_steps=1,
            optim="adamw_8bit",
            output_dir="/tmp/output",
        ),
    )

    trainer.train()

    # Save to volume
    model.save_pretrained(output_path)
    tokenizer.save_pretrained(output_path)
    volume.commit()

    return {"status": "complete", "path": output_path}

@app.local_entrypoint()
def main():
    result = finetune_llama.remote("yahma/alpaca-cleaned")
    print(result)

Chat Template Fine-Tuning

# ChatML format
chat_template = """<|im_start|>system
{system}<|im_end|>
<|im_start|>user
{user}<|im_end|>
<|im_start|>assistant
{assistant}<|im_end|>"""

def format_chat(examples):
    texts = []
    for system, user, assistant in zip(
        examples["system"],
        examples["user"],
        examples["assistant"]
    ):
        text = chat_template.format(
            system=system,
            user=user,
            assistant=assistant
        ) + tokenizer.eos_token
        texts.append(text)
    return {"text": texts}

dataset = dataset.map(format_chat, batched=True)

---

Quick Reference

GPU Requirements by Model Size

Model Size    | 4-bit VRAM | 16-bit VRAM | Recommended GPU
--------------|------------|-------------|------------------
1-3B          | 6-8 GB     | 12-16 GB    | T4, L4
7-8B          | 12-16 GB   | 24-32 GB    | L40S
13B           | 16-20 GB   | 32-40 GB    | L40S, A100
70B           | 40-48 GB   | 140+ GB     | A100 80GB, H100

Unsloth Installation

# Standard installation
pip install "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"
pip install xformers trl peft accelerate

# For CUDA 11.8
pip install "unsloth[cu118-ampere-new] @ git+https://github.com/unslothai/unsloth.git"

# For CUDA 12.1+
pip install "unsloth[cu121-ampere-new] @ git+https://github.com/unslothai/unsloth.git"

Key Hyperparameters

# LoRA Configuration
r = 16                    # Rank: 8-64 (higher = more params, better quality)
lora_alpha = 16           # Usually same as r
lora_dropout = 0          # Unsloth optimized for 0
target_modules = [        # All attention + MLP for best results
    "q_proj", "k_proj", "v_proj", "o_proj",
    "gate_proj", "up_proj", "down_proj"
]

# Training Configuration
batch_size = 2            # Per device (increase for larger GPUs)
gradient_accumulation = 4 # Effective batch = batch_size * accumulation
learning_rate = 2e-4      # Standard for LoRA
warmup_steps = 5          # 5-10% of total steps
max_seq_length = 2048     # Model's max (512-4096)

Common Commands

# Enable inference mode
FastLanguageModel.for_inference(model)

# Save adapter only
model.save_pretrained("path")

# Save merged 16-bit
model.save_pretrained_merged("path", tokenizer, save_method="merged_16bit")

# Save merged 4-bit (smallest)
model.save_pretrained_merged("path", tokenizer, save_method="merged_4bit")

# Push to Hub
model.push_to_hub_merged("user/model", tokenizer, save_method="merged_16bit")

---

Anti-Patterns

❌ Using standard Transformers for fine-tuning: Slower and uses more memory ✅ Use Unsloth for 2-5x speedup with same results

❌ Loading full precision models: Wastes VRAM on consumer GPUs ✅ Always use load_in_4bit=True for models >7B

❌ Wrong dtype for GPU architecture:

# ❌ Bad: Forces FP16 on Ampere+ GPUs
args = TrainingArguments(fp16=True, bf16=False)

# ✅ Good: Auto-detect best dtype
args = TrainingArguments(
    fp16=not torch.cuda.is_bf16_supported(),
    bf16=torch.cuda.is_bf16_supported()
)

❌ Tiny batch sizes without gradient accumulation: Unstable training ✅ Use gradient_accumulation_steps to increase effective batch size

❌ Too many LoRA target modules: Wastes memory for marginal gains ✅ Start with attention layers only: ["q_proj", "k_proj", "v_proj", "o_proj"]

❌ Ignoring sequence length: OOM errors or wasted compute ✅ Set max_seq_length to actual data length (analyze dataset first)

❌ Not using gradient checkpointing: Runs out of VRAM ✅ Always use use_gradient_checkpointing="unsloth" for large models

❌ Saving full model every epoch: Wastes disk space ✅ Save LoRA adapter only (~100MB) or push merged model once at end

❌ Using standard AdamW: Higher memory usage ✅ Use optim="adamw_8bit" for 2x memory reduction

---

Related Skills

• lora-peft-techniques.md - LoRA configuration, rank selection, merging
• llm-dataset-preparation.md - Dataset formatting, quality, validation
• modal-gpu-workloads.md - GPU selection, cost optimization, deployment
• huggingface-autotrain.md - Alternative no-code fine-tuning approach
• llm-inference-optimization.md - Post-training inference optimization
• model-evaluation.md - Evaluating fine-tuned model quality

---

Last Updated: 2025-10-18 Format Version: 1.0 (Atomic)