Every so often a phrase takes root in software development and becomes cargo cult. "We need 80% automation coverage" is one of those phrases. It sounds measurable, ambitious, and modern. It gets repeated in job specs, sprint retrospectives, and engineering blog posts. Nobody seems entirely sure where it came from. Everyone treats it like a law of physics.
In most industries, chasing 80% automation is a reasonable heuristic. In regulated industries — online gaming, FinTech, insurance — it's a statement that will not survive contact with your compliance team.
Where the 80% rule came from
The closest traceable origin is a rough rule of thumb associated with the testing pyramid: automate the stable, repetitive, and predictable layers; keep manual effort for the volatile, exploratory, and judgment-heavy areas. Over time, "automate what you can" morphed into "automate 80%", which morphed into a target, which morphed into a KPI. Like a lot of software wisdom, the original nuance evaporated in translation.
There is nothing in ISTQB's Foundation Level syllabus, nor in the more advanced specialist certifications, that specifies an automation coverage percentage as a quality target. What the ISTQB syllabus does say — and this is worth pausing on — is that ISTQB FL test automation is a tool to support testing, not a replacement for it. The distinction matters enormously in a compliance context.
"Test automation cannot replace manual testing. It supports it."
— ISTQB Foundation Level Syllabus, testing principles
What regulators actually care about
Let's be precise about what "regulated industry" means in practice for QA. In online gaming, the relevant body is typically a gaming laboratory operating under GLI-19 (Gaming Laboratories International standard 19) or equivalent. In financial services, you have the FCA, PRA, or equivalents depending on jurisdiction. In insurance, Solvency II frameworks govern. None of these frameworks evaluate your CI/CD pipeline. None of them ask for your Playwright coverage report.
What they ask for is human attestation. Someone qualified, accountable, and traceable must have reviewed the specific behaviours in question and signed their name to it. When an auditor asks "how do you know the RNG produces statistically fair outcomes?" — your test runner cannot raise its hand. A named tester, with a documented test plan, executing a structured test case against a specific build, producing an evidence pack — that's what they need.
GLI-19 Section 4 specifies that RNG testing must include documented human-led verification of statistical distribution. Automated tests that validate API responses are supporting evidence at best. They do not substitute for the mandated human test execution.
What ISTQB says (and what it doesn't)
The ISTQB Advanced Technical Test Analyst syllabus defines several conditions under which test automation is appropriate: ISTQB ATTA repeatable tests, stable requirements, high-volume data-driven scenarios, regression suites. It is equally clear about conditions where automation adds limited value: exploratory testing, usability testing, and tests that require human judgment about subjective qualities.
Compliance testing often requires exactly the latter. Determining whether a responsible gambling pop-up is "clearly visible and unambiguous" is not a binary assertion your test runner can evaluate. Assessing whether a bonus terms display is likely to mislead a reasonable consumer is a judgment call. These aren't technical oversights in the testing framework — they're fundamental limitations of automated assertion.
The real problem with the metric
The 80% automation target has a deeper problem beyond compliance: it measures the wrong thing. Code coverage percentage measures how much of your code a test suite touches. It says nothing about whether those tests are meaningful, whether they test the right behaviours, or whether they would actually catch a regression. A suite that touches 80% of your codebase with trivial assertions provides less assurance than a suite with 40% coverage focused on your most critical user journeys.
ISTQB FL The ISTQB Foundation Level principle of "testing shows the presence of defects, not their absence" is directly relevant here. An automated suite that passes is evidence that certain specified conditions hold under certain specified inputs. It is not evidence that the software is correct. It is not evidence that you've found all the important bugs. And in a regulated context, it is absolutely not a substitute for documented human verification.
What happens in practice when a team chases the 80% target? They automate tests that shouldn't be automated. They write shallow assertions to inflate coverage numbers. They neglect the exploratory and scenario-based testing that actually finds interesting defects. And then, when an audit arrives or an incident occurs, they discover that their impressive coverage report cannot answer any of the questions being asked.
A better question to ask
Instead of "what percentage of our tests are automated?", ask three different questions:
- What are our highest-risk test scenarios, and how are each of them being executed? Some should be automated. Some must be manual. Some need both.
- What evidence would our compliance team and auditors need if they reviewed our QA process tomorrow? If you can't answer this, the 80% figure is irrelevant.
- What does our test failure data tell us about where bugs actually come from? Automate where regressions happen repeatedly. Apply human judgment where defects require context to interpret.
This is the risk-based testing approach that ISTQB's Advanced syllabi advocate. ISTQB AL Testing effort should be proportional to the probability and impact of failure — not distributed according to a coverage percentage target.
Verdict
The 80% automation target isn't inherently wrong — it's context-dependent guidance that has escaped its context. In a SaaS product with no regulatory obligations, chasing high automation is generally sensible. In a regulated industry, applying it uncritically will leave you with an impressive-looking pipeline and a compliance gap your auditors will find before you do.
The honest answer to "how much should we automate?" is: as much as makes sense given your risk profile, your stability level, and what your regulatory obligations require in terms of human verification. That answer is harder to put on a dashboard. It is also the right answer.
References: ISTQB Foundation Level Syllabus v4.0; ISTQB Advanced Technical Test Analyst Syllabus; ISTQB Advanced Level Test Manager Syllabus; GLI-19 Technical Standard for Online Gaming Systems v1.1.