dspy-setup

Installation, configuration, and language model setup for DSPy framework including Modal and HuggingFace

DSPy Setup

Scope: Installation, LM configuration, environment setup, Modal/HuggingFace integration Lines: ~420 Last Updated: 2025-10-25

When to Use This Skill

Activate this skill when:

• Starting a new DSPy project from scratch
• Configuring language models for DSPy programs
• Setting up Modal-hosted models for DSPy
• Integrating HuggingFace models (Inference API or self-hosted)
• Troubleshooting DSPy installation issues
• Switching between different LM providers (OpenAI, Anthropic, Modal, HF)
• Understanding DSPy's core architecture and initialization

Core Concepts

DSPy Programming Model

Philosophy: Programming, not prompting

• Write declarative Python code instead of brittle prompt strings
• Define task structure with signatures (input/output specs)
• Compose reusable modules (Predict, ChainOfThought, ReAct)
• Let DSPy optimize prompts and weights automatically

Key insight: DSPy separates "what" (signatures) from "how" (modules) from "optimization" (teleprompters)

Installation and Dependencies

Core package:

# Latest stable release
pip install dspy-ai

# Or with uv (recommended)
uv add dspy-ai

# Development/latest version
pip install git+https://github.com/stanfordnlp/dspy.git

Optional dependencies:

# For specific LM providers
uv add openai anthropic cohere litellm

# For HuggingFace
uv add huggingface-hub transformers

# For Modal integration
uv add modal

# For RAG and vector search
uv add chromadb weaviate-client qdrant-client

# For evaluation and metrics
uv add datasets evaluate

Language Model Configuration

LM object: Central to DSPy workflow

• Configure once, use everywhere
• Supports multiple providers (OpenAI, Anthropic, Modal, HuggingFace, local)
• Handles API keys, rate limits, caching

Three configuration patterns:

1. Global default: dspy.configure(lm=lm) - all modules use this LM
2. Module-specific: predictor = dspy.Predict(signature, lm=custom_lm)
3. Context manager: Temporarily override LM for specific operations

---

Patterns

Pattern 1: Basic Setup (OpenAI)

import dspy

# Configure OpenAI LM
lm = dspy.LM(
    model="openai/gpt-4o-mini",
    api_key="your-api-key-here",  # or set OPENAI_API_KEY env var
    max_tokens=1000,
    temperature=0.0,
)

# Set as global default
dspy.configure(lm=lm)

# Test configuration
response = lm("Hello, DSPy!")
print(response)  # Should return LM response

When to use:

• Quick prototyping with OpenAI models
• Production apps with GPT-4 or GPT-3.5
• Cost-effective development with gpt-4o-mini

Pattern 2: Anthropic Claude Setup

import dspy
import os

# Configure Claude
lm = dspy.LM(
    model="anthropic/claude-3-5-sonnet-20241022",
    api_key=os.getenv("ANTHROPIC_API_KEY"),
    max_tokens=4096,
    temperature=0.0,
)

dspy.configure(lm=lm)

# Claude-specific features
lm_with_caching = dspy.LM(
    model="anthropic/claude-3-5-sonnet-20241022",
    api_key=os.getenv("ANTHROPIC_API_KEY"),
    max_tokens=4096,
    # Claude supports prompt caching for long contexts
    cache_prompt=True,
)

When to use:

• Long context windows (200K tokens)
• Advanced reasoning tasks
• Cost optimization with prompt caching

Pattern 3: Modal-Hosted Models

import dspy
import modal

# Deploy model on Modal with GPU
app = modal.App("dspy-llm")

@app.function(
    image=modal.Image.debian_slim().pip_install("vllm", "transformers"),
    gpu="L40S",  # Cost-effective GPU
    timeout=600,
)
@modal.web_endpoint(method="POST")
def inference(request: dict):
    """Modal endpoint for model inference"""
    from vllm import LLM, SamplingParams

    # Load model once (cached across requests)
    llm = LLM(model="meta-llama/Meta-Llama-3.1-8B-Instruct")

    prompt = request.get("prompt")
    params = SamplingParams(
        temperature=request.get("temperature", 0.0),
        max_tokens=request.get("max_tokens", 1000),
    )

    output = llm.generate(prompt, params)
    return {"response": output[0].outputs[0].text}

# Use Modal endpoint in DSPy
# After deploying: modal deploy dspy_modal.py

import requests

class ModalLM:
    """Custom LM wrapper for Modal endpoints"""
    def __init__(self, endpoint_url, max_tokens=1000, temperature=0.0):
        self.endpoint_url = endpoint_url
        self.max_tokens = max_tokens
        self.temperature = temperature

    def __call__(self, prompt, **kwargs):
        response = requests.post(
            self.endpoint_url,
            json={
                "prompt": prompt,
                "max_tokens": kwargs.get("max_tokens", self.max_tokens),
                "temperature": kwargs.get("temperature", self.temperature),
            }
        )
        return response.json()["response"]

# Configure DSPy with Modal endpoint
modal_lm = ModalLM(
    endpoint_url="https://your-app--inference.modal.run",
    max_tokens=1000,
)

# Note: For full DSPy integration, wrap in dspy.LM-compatible class

When to use:

• GPU-accelerated inference without infrastructure management
• Cost-effective deployment (pay only for usage)
• Custom fine-tuned models
• High-throughput batch processing

Benefits:

• Serverless GPU inference (L40S, H100, A100)
• Automatic scaling
• Cold start optimization with Modal's caching

Pattern 4: HuggingFace Inference API

import dspy
import os

# Use HuggingFace Inference API
lm = dspy.LM(
    model="huggingface/meta-llama/Meta-Llama-3.1-8B-Instruct",
    api_key=os.getenv("HUGGINGFACE_API_KEY"),
    api_base="https://api-inference.huggingface.co/models",
    max_tokens=1000,
)

dspy.configure(lm=lm)

# Alternative: Use specific HuggingFace endpoint
lm = dspy.LM(
    model="huggingface/mistralai/Mistral-7B-Instruct-v0.2",
    api_key=os.getenv("HUGGINGFACE_API_KEY"),
    max_tokens=1000,
)

When to use:

• Quick access to HuggingFace hosted models
• Testing different open-source models
• Prototyping without infrastructure setup

Pattern 5: Self-Hosted HuggingFace (vLLM/TGI)

import dspy

# Connect to self-hosted vLLM or Text Generation Inference server
# Assumes you have vLLM running: vllm serve meta-llama/Meta-Llama-3.1-8B-Instruct

lm = dspy.LM(
    model="openai/meta-llama/Meta-Llama-3.1-8B-Instruct",  # vLLM uses OpenAI-compatible API
    api_base="http://localhost:8000/v1",  # vLLM default endpoint
    api_key="EMPTY",  # vLLM doesn't require auth by default
    max_tokens=2000,
)

dspy.configure(lm=lm)

# For Text Generation Inference (TGI)
lm_tgi = dspy.LM(
    model="tgi/meta-llama/Meta-Llama-3.1-8B-Instruct",
    api_base="http://localhost:8080",  # TGI default port
    max_tokens=2000,
)

When to use:

• Running models on your own infrastructure
• Maximum control over model configuration
• High-throughput production deployments

Pattern 6: Modal + HuggingFace Integration

import modal
import dspy

# Deploy HuggingFace model on Modal with optimized serving
app = modal.App("dspy-hf-modal")

@app.function(
    image=modal.Image.debian_slim()
        .pip_install("vllm", "transformers", "torch"),
    gpu="L40S",
    timeout=600,
    # Use Modal volumes for model caching
    volumes={"/models": modal.Volume.from_name("hf-models")},
)
@modal.asgi_app()
def serve():
    """Serve HuggingFace model via vLLM on Modal"""
    from vllm import AsyncLLMEngine, AsyncEngineArgs, SamplingParams
    from vllm.entrypoints.openai.api_server import build_app

    # Configure vLLM engine
    engine_args = AsyncEngineArgs(
        model="meta-llama/Meta-Llama-3.1-8B-Instruct",
        download_dir="/models",  # Use Modal volume
        tensor_parallel_size=1,
    )

    # Build OpenAI-compatible API
    return build_app(engine_args)

# After deployment, use in DSPy
lm = dspy.LM(
    model="openai/meta-llama/Meta-Llama-3.1-8B-Instruct",
    api_base="https://your-app--serve.modal.run/v1",
    api_key="EMPTY",
    max_tokens=2000,
)

dspy.configure(lm=lm)

When to use:

• Best of both worlds: HuggingFace models + Modal's serverless GPUs
• Production deployments with autoscaling
• Cost optimization (only pay when using)

Benefits:

• No infrastructure management
• Automatic GPU scaling
• Model caching with Modal volumes
• OpenAI-compatible API

Pattern 7: Local Models (Ollama)

import dspy

# Run local model via Ollama
# First: ollama pull llama3.2
lm = dspy.LM(
    model="ollama/llama3.2",
    api_base="http://localhost:11434",  # Ollama default port
    max_tokens=2000,
)

dspy.configure(lm=lm)

# No API key needed for local models
# Great for development and privacy-sensitive applications

When to use:

• Offline development
• Privacy requirements (data stays local)
• Cost-free experimentation
• Custom fine-tuned models

Pattern 7.5: Hanzo SDK (Unified Multi-Provider with Auto-Routing)

import dspy
from hanzo import Hanzo

# Initialize Hanzo SDK with hybrid mode (local + cloud)
hanzo = Hanzo(
    inference_mode='hybrid',  # Local first, cloud fallback
    auto_route=True,          # Automatic model selection
    cost_optimize=True,       # Optimize for cost
    
    # Optional: Provider credentials (falls back to env vars)
    node_url='http://localhost:8080',  # Hanzo Node for local
    openai_api_key='sk-...',
    anthropic_api_key='sk-ant-...'
)

# Create DSPy-compatible LM wrapper
class HanzoLM(dspy.LM):
    def __init__(self, hanzo_client, model=None):
        self.hanzo = hanzo_client
        self.model = model
        super().__init__(model=model or "auto")
    
    def __call__(self, prompt, **kwargs):
        response = self.hanzo.chat.completions.create(
            messages=[{'role': 'user', 'content': prompt}],
            model=self.model if self.model else None,  # Auto-route if no model
            **kwargs
        )
        return response.choices[0].message.content

# Configure DSPy with Hanzo
lm = HanzoLM(hanzo, model='llama-3-8b')  # Or model=None for auto-routing
dspy.configure(lm=lm)

# Alternative: Use Hanzo for specific operations
local_lm = HanzoLM(Hanzo(inference_mode='local'), model='llama-3-8b')
cloud_lm = HanzoLM(Hanzo(inference_mode='cloud'), model='gpt-4')

# Privacy-sensitive operations use local
dspy.configure(lm=local_lm)
sensitive_result = predictor(private_data)

# Complex operations use cloud
with dspy.context(lm=cloud_lm):
    complex_result = predictor(complex_query)

When to use:

• Multi-provider flexibility: Switch between local, OpenAI, Anthropic automatically
• Cost optimization: Automatic routing to cheapest suitable model
• Privacy requirements: Local-first with selective cloud fallback
• Production reliability: Automatic failover and retry logic

Benefits over direct provider integration:

• Automatic routing: SDK analyzes query complexity and routes to best model
• Cost tracking: Built-in cost monitoring and budget limits
• Caching: Semantic caching reduces redundant inference
• Privacy-first: Tries local Hanzo Node before cloud providers
• Unified interface: One API for all providers (local + cloud)
• Resilience: Automatic retries, circuit breaking, fallbacks

Comparison: Manual DSPy LM vs Hanzo SDK:

| Feature | Manual DSPy LM | Hanzo SDK + DSPy | |---------|----------------|------------------| | Provider setup | Configure each provider separately | One unified client | | Model routing | Manual selection per query | Automatic based on complexity | | Local inference | Requires separate Ollama/vLLM setup | Built-in via Hanzo Node | | Cost tracking | Custom implementation | Built-in | | Caching | Manual implementation | Automatic semantic caching | | Failover | Custom retry logic | Automatic with fallbacks | | Privacy | Separate local/cloud paths | Local-first automatic routing | | Lines of code | 50+ per provider | 10-15 total |

Example: DSPy Optimizer with Hanzo:

import dspy
from hanzo import Hanzo
from dspy.teleprompt import BootstrapFewShot

# Initialize Hanzo for optimization (use cheaper models)
hanzo_optimizer = Hanzo(
    inference_mode='hybrid',
    routing_strategy='cost_optimized',
    preferred_models=['llama-3-8b', 'gpt-3.5-turbo']
)

lm_optimizer = HanzoLM(hanzo_optimizer)

# Initialize Hanzo for production (use best models)
hanzo_prod = Hanzo(
    inference_mode='hybrid',
    routing_strategy='quality_focused',
    preferred_models=['gpt-4', 'claude-3-opus']
)

lm_prod = HanzoLM(hanzo_prod)

# Define DSPy program
class QA(dspy.Module):
    def __init__(self):
        super().__init__()
        self.generate_answer = dspy.ChainOfThought("question -> answer")
    
    def forward(self, question):
        return self.generate_answer(question=question)

# Optimize with cheap models via Hanzo
dspy.configure(lm=lm_optimizer)
optimizer = BootstrapFewShot(metric=qa_metric)
optimized_qa = optimizer.compile(QA(), trainset=train_data)

# Deploy with best models via Hanzo
dspy.configure(lm=lm_prod)
result = optimized_qa(question="What is quantum computing?")

# Hanzo automatically:
# - Used llama-3-8b for optimization (fast, cheap)
# - Uses gpt-4 for production (high quality)
# - Tracked costs for both phases
# - Cached repeated prompts during optimization

See Also:

• python-sdk.md - Complete Hanzo SDK documentation
• hanzo-node.md - Local inference setup
• llm-model-routing.md - Automatic routing strategies

Pattern 8: Multi-Model Setup

import dspy

# Different models for different purposes
fast_lm = dspy.LM("openai/gpt-4o-mini", max_tokens=500)
smart_lm = dspy.LM("openai/gpt-4o", max_tokens=2000)
modal_lm = dspy.LM(
    "openai/meta-llama/Meta-Llama-3.1-8B-Instruct",
    api_base="https://your-app.modal.run/v1",
    api_key="EMPTY",
)

# Set default for most operations
dspy.configure(lm=fast_lm)

# Use smart model for complex reasoning
class ComplexReasoner(dspy.Module):
    def __init__(self):
        super().__init__()
        # Override with smarter model
        self.predictor = dspy.ChainOfThought("question -> answer", lm=smart_lm)

    def forward(self, question):
        return self.predictor(question=question)

# Use Modal model for high-volume, cost-sensitive tasks
high_volume_predictor = dspy.Predict("text -> category", lm=modal_lm)

Benefits:

• Cost optimization (use expensive models only when needed)
• Performance optimization (fast models for simple tasks)
• Flexibility (switch models without code changes)

---

Quick Reference

Installation Commands

# Standard installation
pip install dspy-ai

# With uv (recommended)
uv add dspy-ai

# With Modal and HuggingFace support
uv add dspy-ai modal huggingface-hub transformers

# Optional dependencies
uv add openai anthropic cohere litellm chromadb weaviate-client

LM Configuration Patterns

| Provider | Model String | API Base | Use Case | |----------|--------------|----------|----------| | OpenAI | openai/gpt-4o-mini | Default | Fast, cost-effective | | OpenAI | openai/gpt-4o | Default | Advanced reasoning | | Anthropic | anthropic/claude-3-5-sonnet-20241022 | Default | Long context, caching | | Modal | openai/model-name | Custom endpoint | Serverless GPU | | HuggingFace | huggingface/model-name | HF Inference API | Hosted open models | | vLLM | openai/model-name | http://localhost:8000/v1 | Self-hosted | | Ollama | ollama/llama3.2 | http://localhost:11434 | Local, privacy |

Modal Deployment Quick Start

# Install Modal
uv add modal

# Set up Modal account
modal setup

# Deploy model endpoint
modal deploy dspy_modal.py

# Get endpoint URL
modal app list

Configuration Checklist

✅ DO: Store API keys in environment variables
✅ DO: Set reasonable max_tokens limits
✅ DO: Use temperature=0.0 for deterministic outputs
✅ DO: Configure global LM with dspy.configure()
✅ DO: Test LM configuration before building modules
✅ DO: Use Modal volumes for model caching
✅ DO: Use L40S GPU for cost-effective Modal deployments

❌ DON'T: Hard-code API keys in source code
❌ DON'T: Use very high temperatures (>0.7) for structured tasks
❌ DON'T: Forget to set max_tokens (can cause unexpected costs)
❌ DON'T: Skip error handling for API calls
❌ DON'T: Leave Modal dev apps running (use modal app stop)

---

Anti-Patterns

❌ Hard-coded API keys: Security risk, not shareable

# Bad
lm = dspy.LM("openai/gpt-4o-mini", api_key="sk-1234567890")

✅ Use environment variables:

# Good
import os
lm = dspy.LM("openai/gpt-4o-mini", api_key=os.getenv("OPENAI_API_KEY"))

❌ No max_tokens limit: Unexpected costs, slow responses

# Bad
lm = dspy.LM("openai/gpt-4o")  # No limit

✅ Set reasonable limits:

# Good
lm = dspy.LM("openai/gpt-4o", max_tokens=2000)

❌ Forgetting dspy.configure(): Modules won't know which LM to use

# Bad
lm = dspy.LM("openai/gpt-4o-mini")
predictor = dspy.Predict("text -> label")  # No LM configured!

✅ Configure globally or per-module:

# Good
lm = dspy.LM("openai/gpt-4o-mini")
dspy.configure(lm=lm)
predictor = dspy.Predict("text -> label")  # Uses configured LM

❌ Not using Modal volumes for model caching: Slow cold starts

# Bad - Downloads model on every cold start
@app.function(image=image, gpu="L40S")
def serve():
    llm = LLM(model="meta-llama/Meta-Llama-3.1-70B")  # Re-downloads every time

✅ Use Modal volumes:

# Good - Cache model in volume
@app.function(
    image=image,
    gpu="L40S",
    volumes={"/models": modal.Volume.from_name("hf-models")},
)
def serve():
    llm = LLM(model="meta-llama/Meta-Llama-3.1-70B", download_dir="/models")

---

Related Skills

• dspy-signatures.md - Next step: Define input/output signatures
• dspy-modules.md - Building composable prediction modules
• dspy-optimizers.md - Optimizing prompts with teleprompters
• modal-functions-basics.md - Modal deployment fundamentals
• modal-gpu-workloads.md - GPU selection and optimization for Modal
• modal-web-endpoints.md - Creating Modal web endpoints for inference
• llm-dataset-preparation.md - Preparing training data for optimization

---

Last Updated: 2025-10-25 Format Version: 1.0 (Atomic)