All work
AI QAPlaywrightAgentsTest automationCost control

AI-powered QA: 3,300 tests, every night

A regression suite of over 3,000 tests kept alive by AI agents. Agents tell flaky or outdated tests apart from real product bugs, repair the tests, and file evidence-backed bug reports.

Context
Internal platform at a B2B marketplace
My role
Owner of the QA phase and bug triage
When
2026
end-to-end tests run every night
3,300+
real bugs filed in the first ~5 weeks
300+
suite growth in its first month
~4×

The challenge

End-to-end tests are valuable but expensive to keep alive. They break whenever the UI changes, so teams stop trusting them, and eventually stop running them.

How it works

  • Fresh environments every night. Sandboxes are built from the latest code on dedicated runners, then the full Playwright suite runs.
  • Flakiness filter. Every failure is re-run on an idle environment before anyone looks at it.
  • AI only where judgement is needed. A headless agent session looks at each confirmed failure and decides: is the test outdated, or is the product broken? Outdated tests get a fix PR. Real bugs go to Jira with screenshots, steps and severity, and the test is quarantined until the fix lands.
  • Bug hunting and coverage. Separate nightly agents explore the product across several “lanes”, such as permissions, money flows and sessions, and extend coverage where it’s thin. A change-impact hook maps merged PRs to the test groups they affect.
  • Morning summary in Microsoft Teams before 9:00.

Design principle: deterministic execution costs no AI — the model only enters per failure. That keeps the nightly cost low and the results reproducible.

Results

The suite grew about 4× in its first month and now runs over 3,300 tests nightly. In the first five weeks it filed 300+ genuine bugs. More than a third were rated High or Blocker. They included permission and money-handling issues that humans had missed.

Your next move

Let’s make
it work.

Tell me about a task that takes too much time, a workflow you want to improve, or an AI solution you have in mind. You don’t need a technical brief.

Book a 15-minute intro

Free, 15 minutes. We’ll discuss your business and a useful next step.
English or Czech · Remote or in Prague

Prefer to write?

I reply within one working day.