Why Auto-Generating Test Cases From Requirements Isn't Enough

The promise and limits of automated test generation

A quality assurance engineer set out to automate one of software testing's most repetitive tasks: converting acceptance criteria into structured test cases. The initial prototype worked surprisingly well, generating test steps and expected results that looked professional enough to copy into a test-management system.

But that's when the real problems emerged.

According to a detailed account published on HackerNews, the engineer quickly realized that generating text that looks like a test case is fundamentally different from designing tests that actually cover risk. The prototype could turn "Given a registered user enters valid credentials, then the user should be logged in" into a formatted test scenario. What it couldn't do was ask the questions an experienced tester would immediately raise: What happens with an incorrect password? What if the account is locked? Are there rate limits on failed attempts?

Why it matters

As AI-powered development tools proliferate, this experiment highlights a critical distinction for engineering leaders: automation can accelerate documentation and formatting, but the reasoning that makes testing valuable—understanding failure modes, prioritizing risk, challenging incomplete requirements—still requires human judgment. Organizations investing in test automation should focus on tools that surface gaps and support decision-making rather than simply maximizing output volume.

Volume versus coverage

The engineer's first instinct was to generate more scenarios when one wasn't enough. That created a new problem: four test cases that all verified the same happy-path behavior with slightly different wording. More output created the appearance of coverage without actually providing it.

The solution was to explicitly categorize scenarios as positive (intended behavior), negative (invalid conditions), or edge cases (boundaries and unusual combinations). This forced the generator to produce genuinely different tests rather than variations on a theme.

Bad requirements produce convincing tests

Perhaps the most dangerous discovery was that the generator could confidently produce professional-looking test cases from vague requirements by filling in missing details. Given "the system should process eligible customers correctly," it might generate an expected result like "transaction should be declined and error message displayed"—even though the original requirement never specified what "process" meant or what should happen.

The engineer added a requirement analysis step to flag missing information—actor, action, expected behavior, valid conditions—before generating tests. The goal wasn't to validate requirements but to make gaps visible rather than silently completed.

The architecture that emerged

What started as a simple "requirement in, test cases out" tool evolved into a pipeline: normalize input, parse criteria, analyze completeness, expand into scenario categories, build expected results, maintain traceability back to source requirements, and export to Excel or JSON.

The engineer built a Streamlit interface not as a product but as a testing tool—a way to quickly spot when positive, negative, and edge scenarios looked too similar, or when expected results introduced unsupported assumptions.

What automation can and cannot do

The prototype can reduce repetitive work by turning requirements into structured starting points, expanding common scenario categories, flagging missing information, and maintaining traceability. What it cannot do is answer questions that require context beyond the acceptance criterion: Is this the right business behavior? Which failure would hurt users most? What production incident from last year should become a regression test?

Those questions, the engineer concluded, require experience with the system that doesn't exist in the requirement document.

The experiment was published as an open-source project, with the full account first appearing on HackerNews. The engineer's conclusion: the interesting problem isn't generating more test cases faster, but determining how much of test design—not just test writing—can be automated without hiding the reasoning that makes tests valuable.