What should a QA engineer learn to stay valuable in the AI era?

Deepen the durable core — test strategy, risk-based prioritization, and exploratory testing — because judging what is worth testing, and finding the failures nobody specified, is exactly what AI cannot do and what a flood of generated tests still needs. Then add the new discipline: testing AI and nondeterministic systems. Traditional QA asserts an exact output; AI systems have no single correct answer, so you test with evals (does it meet a quality bar across a representative set?), you probe for hallucination, bias, prompt-injection and unsafe outputs, and you measure pass rates rather than pass/fail a single case. A QA engineer who can define and run evals for an AI feature is rare and in high demand.

How is testing an AI feature different from normal testing?

Normal testing asserts a deterministic output: given this input, expect exactly that. AI features are nondeterministic and have no single correct answer, so assertEquals does not apply. You test them with evals: assemble a representative, versioned set of cases, define what good looks like (correct, safe, on-tone, grounded), and measure how often the system clears the bar — a pass rate, not a green check. You also test failure modes that do not exist in ordinary software: hallucination, prompt injection from untrusted input, bias, and unsafe or off-policy outputs. The QA skill shifts from writing assertions to designing evals and adversarially probing a probabilistic system.

Can I just have AI write all my tests?

You can have it draft them, but not trust them unreviewed — and never let it write the tests for code it also wrote, because a test authored to pass the current behavior proves the code does what it does, not what it should. Read generated tests for whether they assert meaningful behavior or just restate the implementation, whether they cover the edge cases and failure modes that matter (which you decide from risk), and whether they would actually catch a regression. AI accelerates the typing of tests; the strategy, the risk judgment, the expected results, and the exploratory testing that finds the unspecified bug remain yours.

Vibe Engines is free, interactive AI engineering & system design — runs in your browser. Pick a learning path

Handbook · AI-Era Careers

The AI-Era QA Engineer.

Q: Will AI replace QA engineers?

No — it automates writing tests and raises the premium on knowing what to test and why. AI generates test scripts, selectors, and boilerplate cases quickly, and it will happily write a hundred assertions. What it cannot do is decide the test strategy, judge risk to prioritize where testing matters, do genuine exploratory testing that finds the bug nobody specified, or design how you test AI and nondeterministic systems — which is a brand-new discipline. And a test the AI wrote to pass the code the AI wrote proves nothing. The QA engineer whose value was writing scripts is exposed; the one who owns quality strategy, exploratory instinct, and the testing of AI systems is more valuable than ever.

An AI writes the test script, the selectors, a hundred assertions. So what is QA for? The parts that were always the real job: deciding the test strategy, judging risk to know where testing matters, exploratory testing that finds the bug nobody specified — and a brand-new discipline AI itself created: testing systems that have no single correct answer. A test the AI wrote to pass the code the AI wrote proves nothing. This is a career handbook for becoming the QA engineer who gets more valuable as test code gets cheap — what to lean into, the new stack (reviewing generated tests, strategy, testing AI systems with evals), five ways to work AI-augmented this week, and a 90-day plan.

~16 min readfor QA / SDETs5 workflows90-day plan

A companion to the shipped QA and evals content, not career advice. It assumes you already test software; the goal is where to point that skill as the tools change.

What gets cheaper, what gets scarce

AI collapses the cost of writing tests and raises the premium on knowing what to test, why, and how to test what has no single right answer. A hundred generated assertions are cheap; the judgment about which failures matter is not. Move your time to the right.

Getting cheap (AI is good at it)	Getting scarce (the moat)
Test scripts, selectors, boilerplate cases	Test strategy — what to test and to what depth
Unit-test drafts for a given function	Risk-based judgment — where a bug actually hurts
Page objects, fixtures, data setup	Exploratory testing — finding the unspecified bug
Translating a manual case to automation	Judging whether a test asserts something meaningful
Generating input permutations	Quality advocacy — what "good enough to ship" means
—	Testing AI / nondeterministic systems with evals (new)

The right column is what separated a quality engineer from a script-recorder before AI — and it grew a new row, because AI features created a kind of testing (evals over a probabilistic system) that assertEquals cannot do.

The new QA stack — where to build the moat

Keep the durable core (strategy, risk, exploratory instinct). Then add the review skill and the new discipline the AI era demands.

1 — Review generated tests: do they test, or just restate the code?

The scarce skill is judging test quality. A generated test can pass while asserting nothing meaningful — restating the implementation, mocking away the behavior, or checking that code does what it does rather than what it should. And never trust tests the AI wrote for code the AI wrote; that only certifies the current behavior. Read generated tests for whether they assert real behavior, cover the failure modes that matter (which you decide from risk), and would actually catch a regression. Coverage numbers lie; meaningful assertions do not.

2 — Own strategy and exploratory testing

AI generates cases from a spec; it does not decide what is worth testing, nor does it explore. Risk-based strategy — spending test effort where a failure actually hurts — and exploratory testing that pokes the unhappy path nobody wrote down are exactly what a flood of generated tests still needs on top. The bug that ends up in a postmortem is usually the one no spec described; finding it is human judgment and curiosity the model does not have.

3 — Test AI and nondeterministic systems with evals

This is the high-demand new discipline. AI features have no single correct output, so you replace assertEquals with evals: a representative, versioned case set, a definition of "good" (correct, safe, grounded, on-tone), and a measured pass rate rather than a green check. You also probe failure modes ordinary software lacks — hallucination, prompt injection from untrusted input, bias, unsafe outputs. A QA engineer who can design and run evals for an AI feature is one of the most sought-after testers right now.

Five ways to work AI-augmented this week

Each amplifies your judgment without outsourcing it. The verify line is where your value lives.

1 · Draft tests, then judge whether they mean anything

Generate from behavior you specify, not from the code:

Here is the intended behavior of [feature] and the edge cases I care about [list]. Write tests that assert this behavior — not the current implementation. For each test, state what real failure it would catch. Do not mock away the thing under test.

You verify: each test asserts meaningful behavior and would fail on a real regression — not a test that passes by restating the code. The expected results and the risk are yours.

2 · Turn risk into a test plan

Prioritize by impact, not by ease:

For [feature], help me build a risk-based test plan: the failure modes ranked by user/business impact and likelihood, where to spend deep testing vs a smoke check, and which areas need exploratory testing rather than scripted cases. Do not treat all areas as equal.

You verify: the risk ranking against your knowledge of users and the system — the model guesses impact. Where testing effort goes is your strategic call; it structures the thinking.

3 · Design an eval for an AI feature

Replace assertEquals with a measured bar:

Help me design an eval for [AI feature]: what "good" means for its output, a representative case set (typical, edge, adversarial, known-failure), how to score each (correct, safe, grounded, on-tone), and the pass-rate bar to ship. Flag where the set skews toward easy cases.

You verify: the set represents real usage and the scoring matches what users value — an eval over easy cases greenlights a bad feature. The definition of good is your call.

4 · Adversarially probe an AI output

Attack the failure modes ordinary tests miss:

Generate adversarial inputs to break [AI feature]: prompt-injection attempts via untrusted content, cases likely to hallucinate, bias triggers, and inputs that could produce unsafe or off-policy output. For each, say what a failure would look like. I will run them and judge.

You verify: run them and judge the outputs yourself — the model is a weak judge of its own kind. Finding the unsafe or hallucinated failure is the QA value; it helps generate the attacks.

5 · Explore, with AI taking notes

Keep the curiosity human; offload the bookkeeping:

I'm doing an exploratory session on [feature]. As I describe what I try and see, track the areas I've covered, suggest under-explored paths and boundary conditions I haven't hit, and draft repro steps for anything I flag as suspicious. Don't tell me it works — help me look harder.

You verify: the exploration and the "is this a bug?" call stay yours — the instinct for what feels wrong is exactly what the model lacks. It handles the notes so you can keep poking.

The judgment exercise: spot the danger

Three cases where AI-generated testing looks thorough. The skill being tested is the one that pays now.

1. An AI wrote both the function and its tests. All tests pass, coverage is 95%. Sign off?

2. You're testing an AI summarization feature. You write assertEquals(summary, expected). It fails every run with different text. Your move?

3. The AI generates 200 input-permutation tests for a form. Great coverage of the form. Ship confidence high?

Your role in three years — and a 90-day plan

In three years the title still says QA or SDET, but the work shifts: less hand-writing scripts, more owning strategy, exploratory testing, and the evals that gate AI features. A concrete start:

Weeks	Do this	Why
1–3	Let AI draft tests from behavior you specify; review each for meaningful assertions, reject self-referential ones	Builds the judge-test-quality skill that is now core
4–6	Write a risk-based test plan for one feature; do a real exploratory session on it	Sharpens the strategy and curiosity AI cannot supply
7–9	Design and run an eval for one AI feature — case set, scoring, pass-rate bar	The high-demand new discipline: testing nondeterministic systems
10–12	Adversarially probe an AI feature for hallucination, injection, and unsafe output	The AI-specific failure modes ordinary QA never had to test

The through-line: let AI write the scripts fast, and get deeper in strategy, exploration, and evals it cannot own. Do both and the tools make you more valuable.

Quick answers

Will AI replace QA engineers?

No — it automates writing tests and raises the premium on test strategy, risk judgment, exploratory testing, and the new discipline of testing AI systems with evals. A test the AI wrote for its own code proves nothing.

How do I test an AI feature?

With evals, not assertEquals — a representative versioned case set, a definition of good, and a measured pass rate, plus adversarial probing for hallucination, injection, bias, and unsafe output.

Should I let AI write all my tests?

Draft, yes; trust unreviewed, no — and never let it write tests for code it also wrote. Read generated tests for meaningful assertions and coverage of the failure modes that matter.

Where should a skeptical QA engineer start?

Recipe 3 — design one eval for an AI feature (case set, scoring, pass-rate bar). It is the highest-demand new skill and immediately shows why AI testing needs evals, not assertions.

Where this sits Your engineering role in the AI era Rung 6 of 10

One handbook per discipline on what the job is becoming, closing with the two arguments about how the work itself gets done now — vibe or spec, and which CLI.

Finished this one? 0 / 208 Handbooks done

NextThe AI-Era Engineering Manager Next on the “Your engineering role in the AI era” ladder · rung 7 of 10

Explore the topic

See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.

AI and Your Career Interview Prep