Installation

Install llmtest with the provider you need:


pip install pytest-llmtest[openai]       # OpenAI (GPT-5.x, GPT-4.1, o3, o4-mini)
pip install pytest-llmtest[anthropic]    # Anthropic (Claude Opus 4.6, Sonnet 4.6, Haiku 4.5)
pip install pytest-llmtest[ollama]       # Local (Ollama — no API key)
pip install pytest-llmtest[pydantic-ai]  # Pydantic AI agents
pip install pytest-llmtest[all]          # Everything

Package managers

uv


uv add pytest-llmtest[openai]

PDM


pdm add pytest-llmtest[openai]

Poetry


poetry add pytest-llmtest[openai]

Rye


rye add pytest-llmtest --features openai

Conda + pip


conda create -n llmtest python=3.12
conda activate llmtest
pip install pytest-llmtest[openai]

Requirements

Python 3.10+
Pydantic 2.0+

API keys


# OpenAI
export OPENAI_API_KEY=sk-...
 
# Anthropic
export ANTHROPIC_API_KEY=sk-ant-...
 
# Ollama — no key needed, just run the server
ollama serve