Promptfoo

Open source alternative to Braintrust and Mindgard

26.3K stars
MIT
Active this week
Last commit today

Last verified Oct 23, 2026

Secure LLM-based apps with automated prompt evals, red teaming, and vulnerability scanning. MIT licensed, runs locally, CI/CD integration included.

Visit website GitHub26.3K

Read the full breakdownWho it is for, how it works, trade-offs, the quick start and the FAQ

who it's for

Who Promptfoo is for#

AI engineers adding prompt regression testing to CI/CD

Promptfoo integrates with GitHub Actions and other CI systems to run eval suites on every pull request that changes a prompt, model parameter, or RAG retrieval step. Failed evals block merge, so prompt regressions do not ship to production.

Skip if:

Your team ships prompts without a versioning workflow and runs no automated testing today. Setup pays off with volume and change frequency; a team with one prompt changed rarely gets less value from the integration.

Security teams red-teaming production LLM applications

Generates adversarial probes across 50+ vulnerability categories, from injection and jailbreaks to content policy gaps, tailored to the specific application. Threat intelligence from 300,000+ community users adds attack patterns no internal team would write manually.

Skip if:

Your LLM application is an internal tool with no user-facing text input, no sensitive data outputs, and no compliance requirements around AI-generated content.

ML engineers comparing LLM providers before committing

Run the same prompts against GPT, Claude, Gemini, Bedrock, and local models in one eval run and get a scored comparison matrix. Model selection is based on your actual task distribution, not a third-party benchmark that may not match your inputs.

Skip if:

You have already committed to a single provider and have no near-term reason to evaluate alternatives.

DevSecOps teams integrating AI security into existing pipelines

Connects to GitHub, GitLab, Jenkins, and MCP-compatible agent frameworks without requiring a new security platform procurement. Surfaces findings in pull requests as actionable remediation steps, fitting existing DevSecOps workflows.

Skip if:

Your team already has a centralized security scanning platform with LLM vulnerability coverage and an established CI integration you own.

the problem

The problem it solves#

Prompts break silently when AI providers update their models, and teams typically find out when users report problems. Manual testing against a handful of examples gives false confidence. A prompt that passes ten test cases can fail on the eleventh when phrasing shifts or context changes. Regression testing for LLM applications has no established tooling equivalent to unit test frameworks for traditional code.

Red teaming adds a second layer of complexity. Covering injection attacks, jailbreaks, and content policy gaps across an application requires adversarial probes across dozens of vulnerability categories. Generating enough by hand is slow and misses attack patterns that an active community continuously discovers. Teams without automated scanning test against known vectors only, missing newly discovered attack patterns that a 300,000-user community surfaces weekly.

how Promptfoo solves it

How it solves it#

YAML-based eval configuration

Define prompts, providers, and pass/fail criteria in YAML configuration files. Run `promptfoo eval` to get a scored comparison matrix across every model and prompt variant in the config. No cloud account required; runs locally against any supported provider.

Multi-provider model comparison

Compare GPT, Claude, Gemini, DeepSeek, Bedrock, Ollama, and more in a single eval run. Each run produces a side-by-side matrix showing how each model performs on your specific inputs, supporting data-driven model selection rather than relying on third-party benchmarks.

Red teaming with 50+ vulnerability categories

Generates thousands of context-aware adversarial probes targeting injection attacks, jailbreaks, PII leakage, and other vulnerability types. Attack generation is automated and scales beyond what hand-curated test sets cover. Real-time threat intelligence from 300,000+ community users adds emerging attack patterns.

CI/CD and pull request integration

Runs eval and red team scans in GitHub Actions, GitLab CI, and Jenkins pipelines. Security findings appear as inline pull request comments with actionable remediation steps, so regressions and vulnerabilities surface at code review time rather than post-deploy.

Local execution, no data egress

Evaluations run 100% on your own infrastructure. Prompt content never leaves your machine during a local run, which matters for teams handling sensitive data or working under data residency requirements. The self-hosted path has no mandatory cloud component.

MCP and agent framework support

Tests LLM agents through MCP and popular agent frameworks. Covers tool-call behavior, multi-step reasoning failures, and agent-specific attack surfaces that single-turn prompt evals cannot reach. Connects to agent apps on-premise or in the cloud.

strengths · trade-offs

Strengths and trade-offs#

Strengths

MIT license with fully local evaluationThe full eval and red team suite runs under the MIT license with no mandatory cloud component. Teams can run evals on their own infrastructure, fork the codebase, and use it commercially without licensing fees. Unlike managed evaluation platforms, prompt data never leaves your network during a local run.
26,000+ stars and active foundation lab contributorsThe GitHub repo has over 26,000 stars and more than 2,500 forks. Contributors come from OpenAI, Google, Microsoft, and Amazon. The threat intelligence pool draws on 300,000+ active users, which means new attack vectors from the community reach all users automatically.
Declarative config keeps evals accessibleYAML configuration files define the full test suite without custom scripting. Adding a new model to compare or a new red team scenario is a config change rather than a code change, keeping the eval workflow accessible to team members who are not fluent in TypeScript.
PR-level security findingsUnlike eval tools that produce async reports for later review, Promptfoo delivers security findings inline in pull requests with remediation steps. Regressions and vulnerabilities appear at the code review stage, where they can be fixed before merge rather than after deploy.

Trade-offs

-837 open GitHub issuesThe repo carries over 800 open issues across a TypeScript codebase. Teams running evals against less common providers or uncommon model configurations may encounter undocumented edge cases. The active community typically resolves high-priority bugs quickly, but long-tail issues accumulate.
-Red team setup requires configuration effortThe red team engine is powerful but not zero-configuration. Setting up attack flows, connecting to a target application, and tuning vulnerability categories for your specific use case requires time. Teams expecting a one-click security audit will need to invest in the initial config.
-No built-in shared dashboards for self-hosted teamsThe self-hosted path provides a local web viewer for eval results but does not include shared team dashboards, persistent result history with access controls, or collaboration features. Teams that need shared visibility across members must integrate their own storage layer or use the cloud tier.

versus alternatives

Promptfoo vs alternatives#

Promptfoo vs Braintrust

Both Promptfoo and Braintrust address LLM evaluation, helping teams measure prompt quality, compare models, and catch regressions. The key difference is deployment model and cost structure.

Feature	Promptfoo	Braintrust
License	MIT	Proprietary
Self-hosting	Yes	No
Local data execution	Yes	No
CI/CD integration	Yes	Yes
Cost for self-hosters	Free	Paid subscription

Promptfoo is the better fit for teams that run sensitive prompts and need data to stay on their infrastructure, teams with high eval volumes where per-seat or per-call pricing becomes expensive, and developer teams that prefer open-source tooling in their CI/CD stack. Braintrust is a better fit for teams that want a managed service with built-in collaboration, shared eval history, and no infrastructure to maintain.

Promptfoo vs Mindgard

Both Promptfoo and Mindgard address AI red teaming and vulnerability scanning. Promptfoo combines evaluation and security testing in a single open-source CLI; Mindgard is an enterprise-focused security platform for AI applications.

Feature	Promptfoo	Mindgard
License	MIT	Proprietary
Self-hosting	Yes	No
Vulnerability types covered	50+	Enterprise-managed
Threat intelligence source	300,000+ community users	Proprietary
CI/CD native	Yes	Yes
Cost for self-hosters	Free	Enterprise contract

Promptfoo fits developer and security teams that want to own their red teaming tooling, integrate scans into CI/CD without enterprise procurement, and benefit from community-sourced threat intelligence at no cost. Mindgard suits enterprise security programs that need managed deployment, dedicated support, and compliance reporting frameworks that exceed what a self-hosted open source tool provides.

install · quick start

Quick start#

bash

Install via npm, Homebrew, or pip, then initialize a sample eval configuration.
```bash
npm install -g promptfoo
promptfoo init --example getting-started
```

tech stack · detected from GitHub

What it's built on#

Languages: JavaScriptTypeScript
Frameworks: ExpressReact
Infrastructure: AWS
Tooling: esbuildWebpack

frequently asked

FAQ#

Is Promptfoo free to use commercially?

Yes. The open source version is MIT licensed and free to run commercially on your own infrastructure with no licensing fees. A managed cloud tier is available through promptfoo.dev for teams that prefer not to self-host.

Does Promptfoo support providers other than OpenAI?

Yes. The README lists support for OpenAI, Anthropic, Google Gemini, Azure, Bedrock, DeepSeek, Ollama, and more. The eval and red team engines work against any configured provider.

Does using Promptfoo send prompt data to external servers?

Not in the self-hosted path. Evaluations run 100% locally, with no data leaving your machine or network during a local run. The cloud tier routes data through managed servers, but the self-hosted path has no external data egress.

How does Promptfoo integrate with a CI/CD pipeline?

It integrates with GitHub Actions, GitLab CI, Jenkins, and similar systems. You configure an eval or red team run as a pipeline step. Findings appear as pull request comments and the job exits with a non-zero code on failures, which blocks merge when configured as a required check.

Is Promptfoo still open source after joining OpenAI?

Yes. The README states that Promptfoo joined OpenAI but remains open source and MIT licensed. The project continues to accept community contributions and is actively maintained, with the most recent push to the repository recorded on the same day this item was last updated.

also worth a look

Similar open-source tools#

Overmind

Alternative toBraintrust

Turn agent production traces into fine-tuned models you own.

669 GitHub stars
Licence AGPL-3.0

Langfuse

Alternative toLangSmith

Trace and debug LLM prompts while monitoring inference costs

36,091 GitHub stars
Licence MIT

iFixAi

Alternative toZenity

Self-hosted AI agent auditing across 60 trust and safety inspections.

30,695 GitHub stars
Licence Apache-2.0

Bifrost

Alternative toPortkey

Open-source AI gateway with 23+ providers, failover, and MCP support.

8,975 GitHub stars
Licence Apache-2.0

REA (Reverse Engineer Anything)

Featured

Alternative toRevEng.AI

AI-powered reverse engineering for binaries, apps, and assemblies.

345,900 GitHub stars
Licence MIT

Hermes Agent

Featured

Alternative toGrok Bot

Self-hosted AI agent: persistent memory, multi-channel chat, model choice.

255,514 GitHub stars
Licence MIT

by category

Open source tools in these categories#

102

12,712 GitHub stars
Licence AGPL-3.0
#Email

rakazo

Alternative toGrok Bot

AI teammates you own: your keys, your model, your machine.

4,904 GitHub stars
Licence Apache-2.0
#AI Agents

Omnara

Alternative toClaude Managed Agents

Open-source agent deployment API. Self-host or use Omnara Cloud.

2,957 GitHub stars
Licence Apache-2.0
#AI Agents

Feature your project

The projects we add and every new guide. Free, unsubscribe any time.

Find open source alternatives to paid software

Promptfoo vs Braintrust

Both Promptfoo and Braintrust address LLM evaluation, helping teams measure prompt quality, compare models, and catch regressions. The key difference is deployment model and cost structure.

Feature	Promptfoo	Braintrust
License	MIT	Proprietary
Self-hosting	Yes	No
Local data execution	Yes	No
CI/CD integration	Yes	Yes
Cost for self-hosters	Free	Paid subscription

Promptfoo vs Mindgard

Feature	Promptfoo	Mindgard
License	MIT	Proprietary
Self-hosting	Yes	No
Vulnerability types covered	50+	Enterprise-managed
Threat intelligence source	300,000+ community users	Proprietary
CI/CD native	Yes	Yes
Cost for self-hosters	Free	Enterprise contract

Who Promptfoo is for#

AI engineers adding prompt regression testing to CI/CD

Skip if:

Security teams red-teaming production LLM applications

Skip if:

ML engineers comparing LLM providers before committing

Skip if:

DevSecOps teams integrating AI security into existing pipelines

Skip if:

The problem it solves#

How it solves it#

YAML-based eval configuration

Multi-provider model comparison

Red teaming with 50+ vulnerability categories

CI/CD and pull request integration

Local execution, no data egress

MCP and agent framework support

Strengths and trade-offs#

Strengths

Trade-offs

Promptfoo vs alternatives#

Promptfoo vs Braintrust

Promptfoo vs Mindgard

Quick start#

What it's built on#

FAQ#

Similar open-source tools#

Open source tools in these categories#

LLMOps & AI Tooling

Security & Monitoring

Developer Tools

AI & Machine Learning

Find open source alternatives to paid software

Who Promptfoo is for#

AI engineers adding prompt regression testing to CI/CD

Skip if:

Security teams red-teaming production LLM applications

Skip if:

ML engineers comparing LLM providers before committing

Skip if:

DevSecOps teams integrating AI security into existing pipelines

Skip if:

The problem it solves#

How it solves it#

YAML-based eval configuration

Multi-provider model comparison

Red teaming with 50+ vulnerability categories

CI/CD and pull request integration

Local execution, no data egress

MCP and agent framework support

Strengths and trade-offs#

Strengths

Trade-offs

Promptfoo vs alternatives#

Promptfoo vs Braintrust

Promptfoo vs Mindgard

Quick start#

What it's built on#

FAQ#

Similar open-source tools#

Open source tools in these categories#

LLMOps & AI Tooling

Security & Monitoring

Developer Tools

AI & Machine Learning

Find open source alternatives to paid software