Here is an awkward truth about our industry: much of what you pay for in a traditional penetration test isn’t testing.
That isn’t a criticism of the people who do the work. Good testers are scarce and expensive, and the way tests are bought and delivered pushes most of their week away from the part you actually want. If you are about to buy one, it is worth knowing where the days go.
Where the days go
Most engagements are sold as a number of tester days. Look at how those days are usually spent and a familiar shape appears.
Gathering and scanning. The first stretch goes on finding out what is there: running discovery tools and vulnerability scanners, then waiting for them to finish. Scanners are useful and some of this work matters, but a person watching a progress bar is not testing anything.
Writing it up. The last stretch goes on the report. Findings are copied into a document template, screenshots are pasted in, severity tables are formatted, and the whole thing is exported to PDF. A clear report is worth paying for. Assembling one by hand is not the best use of a specialist’s time.
The sliver in the middle. Between the two sits the part you wanted: a skilled person thinking like an attacker, following leads, combining small weaknesses into a real route to your data, and proving that each finding is genuine. On many engagements it is a surprisingly small part of the week.
The UK’s National Cyber Security Centre calls penetration testing “this powerful but expensive operation”. The expense is real. The question is how much of it buys the powerful part.
Source: NCSC: Penetration testing.
Why the sliver is so thin
A human test is always time-bound. The budget buys a fixed number of days, and nobody is sharp for every hour of them. Working through hundreds of scanner results is tiring, and tired people miss things.
So testers prioritise, as they must. They spend their hours where experience says weaknesses usually are, and move quickly past what looks well built. A sign-in page handled by a well-known identity provider should be fine, so it often gets less attention than it deserves. Sometimes the provider is fine and the way it was connected to the application is not.
None of this is negligence. It is arithmetic. The NCSC notes that because a test “cannot be entirely procedural”, its quality is “closely linked to the abilities of the penetration testers involved”, and that testing with little information about the target can leave vulnerabilities “undiscovered in the time allocated for testing”. Time and talent are the constraint, so how they are spent decides what you get.
Source: NCSC: Penetration testing.
What “penetration test” should mean
There is a second problem, and it is one of language.
Ask most buyers what a penetration test is and they describe a test of one web application. That is a valuable thing to buy. But it is not what an attacker does, and it is not all the term should mean.
An attacker sees one connected business. They start with research: what anyone can learn about you from public sources. They look for the easiest way in, whether that is a forgotten system, a web application, a cloud account or a person who can be persuaded to help. Once inside, they find out how far that foothold reaches. If a building is the easiest way in, they will try the door.
The NCSC defines penetration testing as “a method for gaining assurance in the security of an IT system by attempting to breach some or all of that system’s security, using the same tools and techniques as an adversary might”. For most businesses, the system that matters is the whole organisation. An attacker won’t stop at the edge of one application because a statement of work did.
So we use the words carefully:
- A scan checks the systems you list for known issues. It is fast, cheap and useful, and it is not a penetration test.
- An application test goes deep into one product: who can see what, and how its workflows can be abused. It is one stage of a penetration test.
- A penetration test follows an attack through your organisation, from research to the data that matters, as far as the agreed scope allows.
The distinction matters when you compare quotes. If everything on the market is called a penetration test, the only thing left to compare is price, and a scan will always win on price.
What changes when agents do the volume
AI agents change where the time goes. The work that fills the start and end of a traditional engagement is work agents do at machine speed, in parallel and without tiring: mapping what is exposed, running and re-running tools, trying thousands of possibilities, capturing evidence and drafting the write-up. An agent doesn’t skip the single sign-on integration because it is probably fine. It can check it, along with everything else in scope.
That moves people to the middle, where they belong: steering the agents, confirming every finding, combining weaknesses into attack paths and rating each one for your business.
It also brings new problems, and a good provider should be open about them:
- Consistency. AI is not deterministic, and two runs can find different things. Coverage should be recorded so you can see what was tested and what wasn’t, and a retest should re-run the original scenarios rather than start afresh.
- Bugs versus vulnerabilities. Agents find a lot, and on a well-defended system they can overrate what they find. A finding matters if an attacker could use it to harm your business, and judging that takes a person and your context. A feature that calls a paid third-party service on every request can look harmless to an outsider and matter a great deal to the business paying the bill.
- Marking your own homework. More fixes are now written with AI too, and a fix written to stop one exploit can be too narrow to stop the next. The retest should be independent, check against the original evidence and look for ways around the fix.
What we test today, and what’s coming
We would rather say plainly what we cover than let a broad promise do the work.
Every engagement starts with research into what public sources reveal about your business. From there we test what you expose to the internet: systems, web applications and APIs, cloud platforms, business cloud apps such as Microsoft 365, and AI features.
Internal network and Active Directory testing, and social engineering, are in early access: we are developing them with a small number of pilot customers, each scoped and priced individually. We don’t test physical security, and we will tell you so when we scope.
Each stage, and where it stands, is on What we test.
Seven questions to ask before you buy
Whoever you buy from, these questions show what you are actually paying for:
- Does “penetration test” in this proposal mean one application, or our organisation?
- Does the work start with research into what anyone can learn about us?
- How much of the time is hands-on testing, and how much is scanning, waiting and writing?
- Who confirms each finding, and how is severity rated for our business rather than in general?
- How will we see what was tested and what wasn’t?
- If AI is used, how do you keep results consistent between runs?
- What does the retest check: the original finding only, or ways around the fix?
A good provider will answer every one of them without hesitating.