Skip to main content

TestivAI vs. toHaveScreenshot()

Playwright ships visual comparison in the box. If you already use expect(page).toHaveScreenshot(), this page explains — honestly — what TestivAI adds and when it isn't worth the dependency.

The short answer​

If you test only with Playwright and a pixel diff answers your question, use the built-in. It's free, it's maintained by the Playwright team, it has no extra dependency, and it's genuinely good. We'd rather you use it than install something you don't need.

TestivAI earns its place when the pixel diff stops being the answer:

  • when you need to know why something changed, not just that it did
  • when a one-pixel layout shift fails forty snapshots and you have to eyeball every one
  • when approving a baseline should be a reviewed decision, not a local flag
  • when Playwright isn't your only framework
  • when an AI agent, not a human, is reading the result

What the built-in does well​

Real credit where it's due — this is a well-built feature:

StabilityTakes repeated screenshots until two consecutive ones match before comparing. This is genuinely better than a single capture, and TestivAI does not do it.
Sensible defaultsanimations: "disabled", caret: "hide", threshold: 0.2 (YIQ perceived colour difference) — the common flake sources are off by default.
Fails where it happensIt's an assertion, so the failure lands on the exact test, participates in Playwright's retry logic, and shows Expected / Actual / Diff in the HTML report.
Per-environment baselinesFiles are stored as {test}.spec.ts-snapshots/{name}-{browser}-{platform}.png, so a macOS baseline never fights a Linux one.
Maskingmask: [locator] paints regions #FF00FF before comparing. stylePath injects CSS to hide volatile elements.
Zero installIt's already in @playwright/test.

If that list covers your needs, stop here. Everything below is about problems it doesn't try to solve.

Where TestivAI is different​

1. It tells you why the pixels changed​

A pixel diff says 0.4% of pixels differ. That's where the built-in's job ends and yours begins.

TestivAI captures the DOM and an element map alongside the screenshot, so the report answers the next question for you:

  • DOM identical, pixels differ → render noise (font hinting, anti-aliasing), flagged as such instead of demanding review
  • DOM identical, computed styles differ → a style-only change, with the changed properties listed
  • DOM differs → a structural change, with added/removed/attribute/text counts
  • Each diff region is attributed to the CSS selector that produced it

The verdict reads like "Style-only change — 34 elements restyled with identical DOM (body, header, …). Confirm it's intended before approving."

2. A page shift doesn't fail everything below it​

Add one line to a header and every element under it moves down a pixel. To a pixel differ, that is dozens of independent failures.

TestivAI detects that the elements are the same elements, moved, reports it as a single page shift, and — with shiftTolerance set — passes them. The moved content is still shown in the report, marked as shifted rather than changed.

3. Approval is a reviewed decision​

The built-in updates baselines with --update-snapshots, which rewrites everything the run touched. It's fast, and it's all-or-nothing: an unintended regression captured in the same run gets blessed alongside the intended change.

TestivAI's flow keeps a human in the loop:

  • CI posts the diff on the pull request and uploads the report
  • a reviewer comments /testivai approve <name> (or --all)
  • the action verifies the commenter has write access, then commits the new baselines back to the branch

The approval is attributable, reviewable, and lands in git history like any other change.

4. It notices when a baseline stops being checked​

Delete or rename a test and its baseline silently stops guarding that page. No pixel differ can detect this, because nothing was compared.

TestivAI tracks baselines that received no capture, lists them in the report, and exits 3 by default. That default is deliberate — silent coverage loss is the failure mode nobody catches on their own.

5. One baseline set across frameworks​

If your suite is Playwright plus Selenium, or a Python or Java lane, the built-in covers only the Playwright part. TestivAI's adapters all read and write the same .testivai/baselines/ directory and produce one report.

6. Machine-readable, for agents​

Every run writes visual-report/results.json under a semver-governed schema, and @testivai/mcp exposes it to any MCP client — including explain_snapshot, which hands a model the layered evidence for one snapshot so it can explain the change in your own words. The model is yours; we ship the evidence, not the inference.

Side by side​

toHaveScreenshot()TestivAI
Pixel diff✅✅
Retry until stable✅— (freezes animations, hides caret, waits for fonts)
Fails the individual test✅— (run-level report + exit code)
Per-browser / per-OS baselines✅ automatic✅ via project suffix and {platform} token
Masking✅✅ (hatched and listed in the report)
DOM / style diff — why it changed—✅
Region → selector attribution—✅
Page-shift detection—✅
Noise vs. real change verdict—✅
Review-then-approve in a PR—✅ (write access verified)
Missing-baseline detection—✅ (exit 3 by default)
Shared baselines across frameworks—✅
results.json + MCP for agents—✅
Extra dependencynoneone dev dependency

Choosing​

Use the built-in if Playwright is your only framework, you're comfortable reviewing diffs by eye, and --update-snapshots on your own machine is an acceptable way to approve changes. That describes a lot of good test suites.

Reach for TestivAI when you're spending real time asking why did this change, when small layout shifts make diffs noisy enough to ignore, when baseline approval needs to survive review by someone other than the author, or when an agent is doing the first pass.

They can coexist. Nothing stops you keeping toHaveScreenshot() on the handful of components where a strict pixel lock is exactly right, and running TestivAI across the pages where you need the explanation. They use separate baseline directories and won't collide.

→ Getting started · How it works · Masks, regions & tolerances