Promptfoo: LLM evals & red teaming

📦 About this Python package

This is a lightweight wrapper that installs promptfoo via pip. It requires Node.js 20+ and executes npx promptfoo@latest under the hood.

💡 If you have Node.js installed, we recommend using npx promptfoo@latest directly for better performance:

npx promptfoo@latest init
npx promptfoo@latest eval

See the main project for the official npm package.

🐍 Use this pip wrapper when you:

• Need to install via pip for Python-only CI/CD environments
• Want to manage promptfoo with poetry/pipenv/pip alongside Python dependencies
• Work in environments where pip packages are easier to approve than npm

promptfoo is a developer-friendly local tool for testing LLM applications. Stop the trial-and-error approach - start shipping secure, reliable AI apps.

Website · Getting Started · Red Teaming · Documentation · Discord

Installation

Requirements

• Python 3.9+ (for this wrapper)
• Node.js 20+ (required to run promptfoo)

Install from PyPI

pip install promptfoo

Alternative: Use npx (Recommended)

If you have Node.js installed, you can skip the wrapper and use npx directly:

npx promptfoo@latest init
npx promptfoo@latest eval

This is faster and gives you direct access to the latest version.

Quick Start

# Install
pip install promptfoo

# Initialize project
promptfoo init

# Run your first evaluation
promptfoo eval

See Getting Started (evals) or Red Teaming (vulnerability scanning) for more.

What can you do with Promptfoo?

• Test your prompts and models with automated evaluations
• Secure your LLM apps with red teaming and vulnerability scanning
• Compare models side-by-side (OpenAI, Anthropic, Azure, Bedrock, Ollama, and more)
• Automate checks in CI/CD
• Review pull requests for LLM-related security and compliance issues with code scanning
• Share results with your team

Here's what it looks like in action:

It works on the command line too:

It also can generate security vulnerability reports:

Why Promptfoo?

• 🚀 Developer-first: Fast, with features like live reload and caching
• 🔒 Private: LLM evals run 100% locally - your prompts never leave your machine
• 🔧 Flexible: Works with any LLM API or programming language
• 💪 Battle-tested: Powers LLM apps serving 10M+ users in production
• 📊 Data-driven: Make decisions based on metrics, not gut feel
• 🤝 Open source: MIT licensed, with an active community

How This Wrapper Works

This Python package is a thin wrapper that:

1. Checks if Node.js is installed
2. Executes npx promptfoo@latest <your-args> (or uses globally installed promptfoo if available)
3. Passes through all arguments and environment variables
4. Returns the same exit code

The actual promptfoo logic runs via the official TypeScript package from npm. All features and commands work identically.

Python-Specific Usage

With pip

pip install promptfoo
promptfoo eval

With poetry

poetry add --group dev promptfoo
poetry run promptfoo eval

With requirements.txt

echo "promptfoo" >> requirements.txt
pip install -r requirements.txt
promptfoo eval

In CI/CD (GitHub Actions example)

- name: Setup Node.js
  uses: actions/setup-node@v4
  with:
    node-version: "20"

- name: Install promptfoo
  run: pip install promptfoo

- name: Run red team tests
  run: promptfoo redteam run

Learn More

• 📚 Full Documentation
• 🔐 Red Teaming Guide
• 🎯 Getting Started
• 💻 CLI Usage
• 📦 Main Project (npm)
• 🤖 Supported Models
• 🔬 Code Scanning Guide

Troubleshooting

"ERROR: promptfoo requires Node.js"

The wrapper needs Node.js to run. Install it:

• macOS: brew install node
• Ubuntu/Debian: sudo apt install nodejs npm
• Windows: Download from nodejs.org
• Any OS: Use nvm

Slow First Run

The first time you run promptfoo, npx downloads the latest version from npm (typically ~50MB). Subsequent runs use the cached version and are fast.

To speed this up, install promptfoo globally:

npm install -g promptfoo

The Python wrapper will automatically use the global installation when available.

Version Pinning

By default, this wrapper uses npx promptfoo@latest. To pin a specific version:

export PROMPTFOO_VERSION=0.95.0
promptfoo --version

Or install a specific version globally:

npm install -g [email protected]

Telemetry

This wrapper collects anonymous usage telemetry to help improve the package, mirroring the telemetry in the main promptfoo project.

What is collected: A random anonymous user ID, wrapper version, Python version, and whether you're running in CI. If you've previously logged into promptfoo and your email is stored in ~/.promptfoo/promptfoo.yaml, that is also sent.

To opt out:

export PROMPTFOO_DISABLE_TELEMETRY=1

Contributing

For promptfoo features, docs, and bugs: contribute to promptfoo/promptfoo.

For pip wrapper issues (installation, Python shim, CI integration): open issues here. See CONTRIBUTING.md for development setup.

Join our Discord community for help and discussion.

<a href="https://github.com/promptfoo/promptfoo/graphs/contributors">
<img src="https://contrib.rocks/image?repo=promptfoo/promptfoo" />
</a>

License

MIT License - Same as promptfoo