Promptfoo runs a set of answers through checks you can repeat after a prompt or application changes. Use it when “the reply looks fine” is no longer enough to catch regressions. For a single answer you will never revisit, a manual check is simpler. This guide starts with saved refund replies: you will catch two defects, fix the data and rerun one case, without needing a model API key.
Install on Windows
Use Node.js 22.22.0 or newer; the installation guide recommends Node 24. Open PowerShell in a development folder and install a project-local copy:
node --version
mkdir refund-evals
cd refund-evals
npm init -y
npm install --save-dev promptfoo@0.125.1
npx promptfoo --version
The last command should print 0.125.1. Keep package.json and package-lock.json with your tests so another developer can install the same dependency version.
These examples use Promptfoo as a CLI, so there is no MCP server to register. In Codex, Claude Code or Cursor, open this folder and ask the assistant to edit the YAML and run the same terminal commands. A useful request is: “Keep the deliberately wrong answers, run the refund eval, and explain each failing assertion before changing a rule.”
For this project, keep evaluation state beside the configuration. Run this in each new PowerShell session before the eval commands:
$env:PROMPTFOO_CONFIG_DIR = Join-Path $PWD '.promptfoo-local'
$env:PROMPTFOO_DISABLE_TELEMETRY = '1'
First eval: valid JSON can still contain a bad refund
Save this as promptfooconfig.yaml in refund-evals. A provider supplies the answer to evaluate; here, the official echo provider returns the saved text. An assertion is a rule that decides whether that answer passes.
description: Validate saved refund replies
prompts:
- '{{saved_answer}}'
providers:
- echo
tests:
- description: valid-refund
vars:
saved_answer: '{"category":"refund","amount":120}'
assert:
- type: is-json
- type: javascript
value: JSON.parse(output).category === "refund" && JSON.parse(output).amount > 0
- description: invalid-amount
vars:
saved_answer: '{"category":"refund","amount":-1}'
assert:
- type: is-json
- type: javascript
value: JSON.parse(output).amount > 0
- description: invalid-json
vars:
saved_answer: 'Refund approved'
assert:
- type: is-json
Run the suite and inspect the native command's exit code immediately afterwards:
npx promptfoo eval -c promptfooconfig.yaml -o results.json --no-cache --no-share
$LASTEXITCODE
You should see 1 passed, 2 failed, 0 errors, followed by 100. The CLI worked: it found the two defects we planted. Open results.json in your editor and inspect the individual response and grading results.
| Case | What happens | Why |
|---|---|---|
valid-refund |
Passes | JSON, refund category, positive amount |
invalid-amount |
Fails | -1 is legal JSON but violates the amount rule |
invalid-json |
Fails | Refund approved is a sentence, not JSON |
In our Windows run the failure messages included Custom function returned false and Expected output to be valid JSON. This is why checking JSON syntax alone is insufficient for a refund contract. Also notice that the third case checks only syntax; different cases can enforce different rules.
Three ways to use the suite
1. Catch changes to a response contract
Keep the deliberately bad cases and run them whenever you change the assertions:
npx promptfoo eval -c promptfooconfig.yaml -o contract-results.json --no-cache --no-share
The expected result is still one pass and two failures. If the negative amount suddenly passes, investigate the rule before trusting a better overall score. For real refund data, add checks for required fields, numeric types and permitted categories; the small example above is not a complete accounting schema.
2. Debug one case without running everything
npx promptfoo eval -c promptfooconfig.yaml --filter-pattern '^valid-refund$' -o one-case.json --no-cache --no-share
$LASTEXITCODE
Expect one passing case and exit 0. The anchors in ^valid-refund$ select that entire description, so adding a similarly named test later will not silently expand this run. Keep readable descriptions: they become your handles for investigating regressions.
3. Prove a correction changes the result
Copy promptfooconfig.yaml to fixed.yaml. In that copy, replace the negative amount -1 with 80, and replace Refund approved with {"category":"refund","amount":80}. Preserve the assertions and original file.
npx promptfoo eval -c fixed.yaml -o fixed-results.json --no-cache --no-share
$LASTEXITCODE
Expect 3 passed, 0 failed, 0 errors, then 0. Compare the original and fixed JSON exports: the answers changed, while the rules stayed in place. That distinction is useful when an assistant suggests “fixing” a failing suite by deleting the assertion.
Windows errors we hit, and what to change
“The libsql binding for win32-x64-msvc is missing”
We hit this after using dependencies installed on Linux in a Windows run. The message identified @libsql/win32-x64-msvc as the missing package. Install dependencies on Windows in a fresh project folder using the commands above; move the configuration and lockfile between platforms, rather than an installed node_modules directory. The official troubleshooting page covers other installation methods.
The eval exits 100
Read the summary before reinstalling anything. By default, exit 100 means failed tests, whereas other execution errors use exit 1. In PowerShell, read $LASTEXITCODE directly after the eval; a wrapper script finishing successfully does not tell you whether its native child command passed. PROMPTFOO_FAILED_TEST_EXIT_CODE can change that convention in an existing team's environment.
Valid JSON fails, or an “offline” eval requests a key
For valid JSON, inspect the other assertions: our negative refund failed the amount rule, not parsing. For a key request, inspect both providers and every assertion. Echo passes through saved text, but model-based graders such as llm-rubric can still call a model; the local is-json and JavaScript checks here do not.
An editor terminal uses the wrong Node version
Run node --version inside the terminal executing the eval. If it is below 22.22.0, update that runtime and reopen the terminal before installing again. Checking a different terminal's version does not establish what your coding assistant is running.
Cost, limits and when to use something else
Promptfoo's CLI is MIT-licensed. Echo and these deterministic assertions do not generate model tokens. Connecting a model provider or using a model grader introduces that service's access requirements and potential charges; see the official first-model guide when you are ready to evaluate generation.
Passing this suite establishes only the rules you wrote. It does not establish factual correctness, good judgment or production safety. Review JavaScript assertions before executing someone else's configuration, since they run code.
Use JSON Repair when the job is to repair malformed JSON; Promptfoo helps you detect defects rather than repair each response. Use LangGraph to build an agent workflow; an eval suite can then check that workflow's saved outputs. If you already have a few ordinary deterministic unit tests and no prompt/provider matrix to compare, your existing test runner may be enough.
FAQ
How do I install Promptfoo on Windows?
Install Node 22.22.0 or newer, then run npm install --save-dev promptfoo@0.125.1 inside a project. Confirm the installed version with npx promptfoo --version. Install the dependencies on Windows itself to get its native database binding.
Can I use Promptfoo without an OpenAI API key?
Yes: echo lets you evaluate answers that already exist, using local assertions such as JSON and JavaScript checks. That is useful for validating a response contract before adding live generation. Choosing an API-backed provider or model-based grader changes the requirements.
Why does Promptfoo return exit code 100?
It is the default failed-test exit code. An evaluated answer failed an assertion, or the configured pass-rate threshold was not met; inspect the report to find the case. In this guide, the first run should return 100 because two answers are deliberately wrong.
What should go in a Promptfoo dataset?
Include representative inputs or saved answers, clear case descriptions and explicit requirements. Keep known-bad examples that must fail as well as ordinary passing cases. Otherwise, weakening a rule can make the score improve while making the suite less useful.
What we tested
On November 16, 2026, the team ran Promptfoo 0.125.1 natively on Windows through a Codex CLI task and separately in Linux. The real CLI produced 1 pass/2 failures with exit 100, 3 passes with exit 0 after correction, and a single passing filtered case. The inputs were synthetic saved answers evaluated with echo; no model generation, model grader, red-team scan, hosted sharing or report UI was tested. Official installation, echo and command-line references were checked on the same date.