What you are actually buying when you buy a penetration test

A buyer sends the same request to three firms and gets back quotes at eight thousand, twenty-six thousand and fifty thousand dollars. The words in the three proposals are close to identical. Everyone involved concludes that security pricing is arbitrary, and the decision gets made on price, which is the one input guaranteed not to correlate with what arrives.

The prices are not arbitrary. They are buying different things under the same noun, and the difference is legible in advance. We write this knowing we sell this service, so the useful thing we can do is tell you how to interrogate a quote, including ours.

The first fork: scanning or testing¶

A vulnerability scan and a penetration test are different work, and NIST's technical guide to information security testing (opens in a new tab) has drawn the line between them for a long time. A vulnerability scan enumerates: it compares what it finds against a database of known flaws and reports matches. A penetration test attempts: it tries to exploit what it finds, and it chains findings together, which is the part that matters, because the way real intrusions work is that three unremarkable weaknesses combine into one serious one.

Both are legitimate purchases. A scan is cheap, repeatable, and the right instrument for tracking hygiene across a large estate over time. A test is expensive, is a point-in-time judgment by a person, and is the right instrument for answering whether a determined attacker gets to the thing you care about. The failure is not buying the cheap one; it is buying the cheap one while believing you bought the expensive one, and then telling a customer or an insurer you have been penetration tested.

The tell is in the deliverable's shape. A report whose findings are each a single CVE with a CVSS score, sorted by severity, with generic remediation text, is a scan output that has been formatted. A report from a test contains a narrative: this got us here, which let us do that, which is why the third thing matters more than its score suggests. Ask to see a sanitized example before you sign. Any firm worth hiring has one ready.

The second fork: what the scope actually says¶

Most scopes are written as a list of addresses or applications, because that is what is easy to write and easy to price. What you get back is a test of those things, which is only useful if your question was about those things. It usually was not.

The stronger form is a goal. Starting from the guest wireless, can you reach the file share holding customer drawings? Starting from a phished sales account, can you reach the production controllers? A goal-scoped test can tell you that the answer is yes by a route nobody had listed, which is precisely the finding that a list-scoped test cannot produce, because the route ran through a system that was not on the list.

Goal scoping costs more and it is worth arguing for. It also requires you to say out loud what you are protecting, which some organizations find harder than the technical work.

The four questions¶

1. 1.What happens if you find nothing? A firm that answers “we would tell you, and here is what we would have to have done for that answer to mean anything” is selling an assessment. A firm that cannot imagine the outcome is selling a report with a predetermined length.
2. 2.Who does the work, and can I read something they wrote? Testing quality is a property of individuals rather than of firms. The name on the proposal is often not the name on the keyboard, and asking is not rude.
3. 3.Do I get the raw evidence (requests, responses, screenshots, timestamps) or only the write-up? Without it, a disputed finding cannot be settled and a fix cannot be verified.
4. 4.Is a retest included, and for how long? Findings get fixed weeks later. A test with no retest ends with a list of claims about a system that no longer exists.

What we would tell you not to buy from us¶

A penetration test is the wrong first purchase for a fair number of the organizations that ask us for one, and it is worth saying which. This is the same conversation we open with on our offensive security engagements, usually before a scope exists.

If you already know the answer, do not pay to confirm it. An organization with a flat network, shared administrator credentials and no logging does not need a test to establish that an attacker would succeed; it needs the segmentation, and spending the budget on a report that says so is spending it on documentation of a thing you told us in the first meeting.

If nobody can act on the result, defer it. A test produces work, and work needs an owner and a window. Reports that arrive at organizations with no capacity to remediate become artifacts for an auditor, which is a real use, but it is a compliance purchase and should be priced and scoped as one rather than as a security purchase.

And if the driver is a questionnaire rather than a threat, say so up front, because the cheapest honest thing that satisfies a questionnaire is frequently not a test. The same rule governs it that governs compliance scoping generally: the scope answers the question, and the question came from somebody else's form. Buying a broader test than the form asks for is a decision, and it should be made deliberately rather than by a vendor who was not told which of the two you were doing.

The test¶

Before you send the request out, write one sentence: the thing I am afraid of is someone reaching X by way of Y. If you can write it, put it in the scope and let three firms tell you how they would go after it, and compare those answers rather than the prices. If you cannot write it, that is the piece of work to do first, and it costs nothing but an afternoon with the right four people in a room.