An AI writes the test script, the selectors, a hundred assertions. So what is QA for? The parts that were always the real job: deciding the test strategy, judging risk to know where testing matters, exploratory testing that finds the bug nobody specified — and a brand-new discipline AI itself created: testing systems that have no single correct answer. A test the AI wrote to pass the code the AI wrote proves nothing. This is a career handbook for becoming the QA engineer who gets more valuable as test code gets cheap — what to lean into, the new stack (reviewing generated tests, strategy, testing AI systems with evals), five ways to work AI-augmented this week, and a 90-day plan.
~16 min readfor QA / SDETs5 workflows90-day plan
A companion to the shipped QA and evals content, not career advice. It assumes you already test software; the goal is where to point that skill as the tools change.
01
What gets cheaper, what gets scarce
AI collapses the cost of writing tests and raises the premium on knowing what to test, why, and how to test what has no single right answer. A hundred generated assertions are cheap; the judgment about which failures matter is not. Move your time to the right.
Getting cheap (AI is good at it)
Getting scarce (the moat)
Test scripts, selectors, boilerplate cases
Test strategy — what to test and to what depth
Unit-test drafts for a given function
Risk-based judgment — where a bug actually hurts
Page objects, fixtures, data setup
Exploratory testing — finding the unspecified bug
Translating a manual case to automation
Judging whether a test asserts something meaningful
Generating input permutations
Quality advocacy — what "good enough to ship" means
—
Testing AI / nondeterministic systems with evals (new)
The right column is what separated a quality engineer from a script-recorder before AI — and it grew a new row, because AI features created a kind of testing (evals over a probabilistic system) that assertEquals cannot do.
02
The new QA stack — where to build the moat
Keep the durable core (strategy, risk, exploratory instinct). Then add the review skill and the new discipline the AI era demands.
1 — Review generated tests: do they test, or just restate the code?
The scarce skill is judging test quality. A generated test can pass while asserting nothing meaningful — restating the implementation, mocking away the behavior, or checking that code does what it does rather than what it should. And never trust tests the AI wrote for code the AI wrote; that only certifies the current behavior. Read generated tests for whether they assert real behavior, cover the failure modes that matter (which you decide from risk), and would actually catch a regression. Coverage numbers lie; meaningful assertions do not.
2 — Own strategy and exploratory testing
AI generates cases from a spec; it does not decide what is worth testing, nor does it explore. Risk-based strategy — spending test effort where a failure actually hurts — and exploratory testing that pokes the unhappy path nobody wrote down are exactly what a flood of generated tests still needs on top. The bug that ends up in a postmortem is usually the one no spec described; finding it is human judgment and curiosity the model does not have.
3 — Test AI and nondeterministic systems with evals
This is the high-demand new discipline. AI features have no single correct output, so you replace assertEquals with evals: a representative, versioned case set, a definition of "good" (correct, safe, grounded, on-tone), and a measured pass rate rather than a green check. You also probe failure modes ordinary software lacks — hallucination, prompt injection from untrusted input, bias, unsafe outputs. A QA engineer who can design and run evals for an AI feature is one of the most sought-after testers right now.
03
Five ways to work AI-augmented this week
Each amplifies your judgment without outsourcing it. The verify line is where your value lives.
1 · Draft tests, then judge whether they mean anything
Generate from behavior you specify, not from the code:
Here is the intended behavior of [feature] and the edge cases I care about [list]. Write tests that assert this behavior — not the current implementation. For each test, state what real failure it would catch. Do not mock away the thing under test.
You verify: each test asserts meaningful behavior and would fail on a real regression — not a test that passes by restating the code. The expected results and the risk are yours.
2 · Turn risk into a test plan
Prioritize by impact, not by ease:
For [feature], help me build a risk-based test plan: the failure modes ranked by user/business impact and likelihood, where to spend deep testing vs a smoke check, and which areas need exploratory testing rather than scripted cases. Do not treat all areas as equal.
You verify: the risk ranking against your knowledge of users and the system — the model guesses impact. Where testing effort goes is your strategic call; it structures the thinking.
3 · Design an eval for an AI feature
Replace assertEquals with a measured bar:
Help me design an eval for [AI feature]: what "good" means for its output, a representative case set (typical, edge, adversarial, known-failure), how to score each (correct, safe, grounded, on-tone), and the pass-rate bar to ship. Flag where the set skews toward easy cases.
You verify: the set represents real usage and the scoring matches what users value — an eval over easy cases greenlights a bad feature. The definition of good is your call.
4 · Adversarially probe an AI output
Attack the failure modes ordinary tests miss:
Generate adversarial inputs to break [AI feature]: prompt-injection attempts via untrusted content, cases likely to hallucinate, bias triggers, and inputs that could produce unsafe or off-policy output. For each, say what a failure would look like. I will run them and judge.
You verify: run them and judge the outputs yourself — the model is a weak judge of its own kind. Finding the unsafe or hallucinated failure is the QA value; it helps generate the attacks.
5 · Explore, with AI taking notes
Keep the curiosity human; offload the bookkeeping:
I'm doing an exploratory session on [feature]. As I describe what I try and see, track the areas I've covered, suggest under-explored paths and boundary conditions I haven't hit, and draft repro steps for anything I flag as suspicious. Don't tell me it works — help me look harder.
You verify: the exploration and the "is this a bug?" call stay yours — the instinct for what feels wrong is exactly what the model lacks. It handles the notes so you can keep poking.
04
The judgment exercise: spot the danger
Three cases where AI-generated testing looks thorough. The skill being tested is the one that pays now.
1. An AI wrote both the function and its tests. All tests pass, coverage is 95%. Sign off?
The core AI-testing trap: a model that writes the tests for its own code produces a self-consistent green suite that proves the code does what it does — including its bugs. High coverage measures lines executed, not behavior verified. Judging whether tests assert intended behavior, derived from risk rather than the implementation, is exactly the QA judgment that is now the job.
2. You're testing an AI summarization feature. You write assertEquals(summary, expected). It fails every run with different text. Your move?
Exact-match assertions do not fit nondeterministic AI output — the feature is working as designed and there is no single right summary. Pinning temperature or substring-matching are brittle hacks that miss quality. The AI-era answer is evals: define good (faithful, complete, concise), run a representative set, and measure the pass rate. That shift from assertion to eval is the new discipline.
3. The AI generates 200 input-permutation tests for a form. Great coverage of the form. Ship confidence high?
Two hundred permutations give deep, cheap coverage of what was specified — and say nothing about the failures nobody wrote down, which is where the shipping bugs live. Quantity of generated tests is not quality of testing. Knowing that exploratory and risk-based testing must cover the unspecified space is the judgment that generated tests cannot replace.
05
Your role in three years — and a 90-day plan
In three years the title still says QA or SDET, but the work shifts: less hand-writing scripts, more owning strategy, exploratory testing, and the evals that gate AI features. A concrete start:
Weeks
Do this
Why
1–3
Let AI draft tests from behavior you specify; review each for meaningful assertions, reject self-referential ones
Builds the judge-test-quality skill that is now core
4–6
Write a risk-based test plan for one feature; do a real exploratory session on it
Sharpens the strategy and curiosity AI cannot supply
7–9
Design and run an eval for one AI feature — case set, scoring, pass-rate bar
The high-demand new discipline: testing nondeterministic systems
10–12
Adversarially probe an AI feature for hallucination, injection, and unsafe output
The AI-specific failure modes ordinary QA never had to test
The through-line: let AI write the scripts fast, and get deeper in strategy, exploration, and evals it cannot own. Do both and the tools make you more valuable.
06
Quick answers
Will AI replace QA engineers?
No — it automates writing tests and raises the premium on test strategy, risk judgment, exploratory testing, and the new discipline of testing AI systems with evals. A test the AI wrote for its own code proves nothing.
How do I test an AI feature?
With evals, not assertEquals — a representative versioned case set, a definition of good, and a measured pass rate, plus adversarial probing for hallucination, injection, bias, and unsafe output.
Should I let AI write all my tests?
Draft, yes; trust unreviewed, no — and never let it write tests for code it also wrote. Read generated tests for meaningful assertions and coverage of the failure modes that matter.
Where should a skeptical QA engineer start?
Recipe 3 — design one eval for an AI feature (case set, scoring, pass-rate bar). It is the highest-demand new skill and immediately shows why AI testing needs evals, not assertions.