"Compliant vs. Secure"

Jessy Lang: Welcome back to Security Dialogues. I'm Jessy, and this is episode nine. Just the two of them tonight. A client of Marcus's passed their SOC 2 Type Two renewal this summer, zero findings, clean report, the kind of result that gets a slide in the next board deck. They had cake in the break room. Ten weeks later, Tess got called in after a breach, through a control that report had just certified as operating effectively.

Marcus Hale: The cake was a mistake in hindsight, though I'd have had it again at the time. The report was accurate. That's the part people don't expect me to say.

Tess Calder: The control existed. I watched the breach happen anyway, in real time, in a replay of their logs. Both things were true at once.

Marcus Hale: What actually got you the call, if you don't mind me asking from the other side for once. I only ever hear the version where the phone rings.

Tess Calder: A junior analyst on their side noticed a login pattern that didn't match the account's normal hours, flagged it, and someone had the sense to escalate instead of closing the ticket. That part had nothing to do with the audit either. That was just someone paying attention on an ordinary Tuesday.

Jessy Lang: Marcus, Tess, this is where tonight starts.

— I. The Celebration and the Breach —

Tess Calder: Walk me through the ten weeks. What changed between the clean audit and the night I got the call?

Marcus Hale: Nothing changed, officially. The access-review control that got certified, quarterly review of who has standing access to the production database, had run exactly as documented, on schedule, with the sign-offs filed where the auditor expected to find them. Then, six weeks after the audit closed, a contractor's account that should have been part of that review got provisioned through a new onboarding tool nobody had connected to the review process yet. It sat outside the process the audit had just certified, from the week it was created.

Tess Calder: So the control the report praised was true the day they tested it, and false by the time I needed it to be true.

Marcus Hale: True on the day. The report never claimed anything about the week after.

Tess Calder: Nobody reads it that way, though. I've sat in board meetings where a clean SOC 2 gets presented like a weather forecast for the whole year, not a photograph of one specific Tuesday.

— II. What an Audit Actually Certifies —

Marcus Hale: Let's be precise about what that report actually says, because almost nobody who waves it around could tell you. An auditor picks a sample, not everything, a sample, of instances where a control should have operated during a defined window, usually six or twelve months. They check whether the control existed, whether it was documented, and whether, for the instances they sampled, it actually ran as described. That's the whole claim.

Tess Calder: It's a claim about the past, about a subset, inside a boundary somebody drew.

Marcus Hale: A fairly narrow claim, stated with a lot of confidence, because confidence is what the format rewards. "We reviewed access quarterly throughout the period and found no exceptions" is true, and it says nothing at all about the contractor account that didn't exist yet when the sample was drawn.

Tess Calder: So the report is honest. It's just answering a much smaller question than the one everyone thinks they're asking it.

— III. Stitched Compliance —

Tess Calder: There's an even thinner version of this we haven't touched yet, and it isn't even an audit. What happens when the evidence is just a stack of policy documents somebody hands over, and the person reading them has no mandate to audit anything at all?

Marcus Hale: That's a tender office, not an auditor, and it's exactly the gap a client of mine fell into this year. We were doing vendor security review as part of their procurement process. A bidder submitted a stack of policy documents: information security policy, incident response policy, access control policy, every section a tender checklist asks for, present and accounted for.

Tess Calder: But.

Marcus Hale: But read closely, not one of them held together. The writing itself was the first tell, fluent in places, oddly generic and inconsistent in others, like four different people had written four different sections and never compared notes. Which, as far as I can tell, is close to what actually happened, except the four different people were one model, prompted section by section.

Tess Calder: Give me a concrete inconsistency, not just a vibe.

Marcus Hale: The objectives document, the top-level security goals the whole program was supposedly built to achieve, was approved two hours before the document describing the measures meant to achieve them. Not years apart. Not even drafted in parallel. Two hours. And when I actually mapped the measures against the objectives, they covered less than half of what the objectives claimed to require.

Tess Calder: So the measures weren't derived from the objectives. They were generated near them, with a timestamp slapped on afterward to make it look like sequence.

Marcus Hale: That's exactly what it reads like. And it gets more specific. The policies defined roles beautifully: a security officer, an incident commander, a data protection lead, each with a tidy paragraph of responsibilities. Not one of them was mapped to an actual resource. No named person, no team, no system access tied to the role. The role existed as a concept in a document and nowhere else.

Tess Calder: A role with nothing underneath it is just a job title nobody holds.

Marcus Hale: Which is invisible if you're only checking whether the policy mentions a security officer. It becomes very visible the moment you ask who that is, specifically, today.

Tess Calder: And the tender office signed off on this.

Marcus Hale: Because a tender office isn't an audit body. Its job is to confirm a bidder submitted the required documents, not to verify the documents describe a system that's actually been operated. Paper is endlessly patient. It doesn't object to what's written on it.

Tess Calder: So this is a rung below everything else we've named so far. SOC 2 at least samples real tickets. ISO at least interviews someone. A tender review might do neither.

Marcus Hale: Call it stitched compliance. Not stale, like a control that was true at audit time and drifted afterward. Stitched: individually plausible fragments, assembled, never reconciled against each other, reviewed by someone who had neither the mandate nor the time to check the seams.

Tess Calder: Three different depths of the same problem, then. Audits sample a real system and go stale over time. Policies satisfy a control on paper while the practice behind them quietly never matches. And now: documents that were never operated even once, stitched together well enough to pass a topic checklist, reviewed by nobody positioned to catch it.

Marcus Hale: Three different depths, and the same question missing from all of them: who's actually checking that what's in front of me describes something real, right now, and not just something complete-looking.

— IV. Three Frameworks, Three Snapshots —

Tess Calder: We keep saying "compliance" like it's one thing. It isn't. Walk me through what actually differs between the frameworks we both deal with, because I don't think the snapshot problem is identical across all of them.

Marcus Hale: ISO 27001 certifies that a management system exists: a defined process for identifying risks, choosing controls, and reviewing them on a schedule you set yourself. It's less "these specific controls are strong" and more "you have a disciplined way of deciding what your controls should be." SOC 2 is narrower and more concrete, it attests that a specific set of controls, chosen from a standard catalogue, actually operated as described over a defined period. The Cyber Resilience Act is different in kind from both. It's not a voluntary attestation at all, it's a legal requirement to handle vulnerabilities across a product's entire support life, and it doesn't come with a calm annual renewal date, it comes with regulatory consequences if you fall behind.

Tess Calder: So ISO is a snapshot of your process for deciding things. SOC 2 is a snapshot of specific controls actually running. CRA isn't a snapshot at all, it's a standing obligation that happens to get checked periodically.

Marcus Hale: Which makes CRA the one framework where treating a point-in-time check as a durable fact isn't just a gap, it's closer to a compliance failure in its own right. If your vulnerability handling was staffed properly in March and quietly stopped being staffed by August, that's not merely a stale audit anymore. Depending on how the regulator reads it, that's you no longer meeting a live legal requirement.

Tess Calder: So the stakes for keeping the thing continuously true, not just periodically true, are different depending on which framework you're even talking about.

Marcus Hale: Genuinely different. Which is exactly why lumping them all together as "compliance" hides how differently wrong it can go.

— V. Snapshot Assurance —

Marcus Hale: I want to name this properly, because it's the same shape as something we picked apart last time, just one layer up. Call it snapshot assurance. A point-in-time photograph of a control, treated afterward as if it were a live, current fact instead of a dated one.

Tess Calder: Same family as the CVSS problem. A number computed once, frozen at disclosure, mistaken for something that updates itself.

Marcus Hale: Exactly the same mechanism, wearing different clothes. Last time it was a severity score, computed for nobody in particular, borrowed as if it were about you. This time it's a control, verified on one date, borrowed as if it were still being verified on every date after.

Tess Calder: And the gap between "was true" and "is true" is exactly where I get in.

Marcus Hale: It's not even the exotic gap. It's the most ordinary one there is. Systems change constantly. Audits don't.

Tess Calder: There's a version of this I want to flag before we move on, because I've seen it used the wrong way. "We passed the audit" showing up afterward, in an incident retro, as a kind of alibi. As if the report's existence were itself a defence, instead of a description of one Tuesday.

Marcus Hale: I've sat in exactly that retro. "We were SOC 2 compliant at the time" said in a tone that was clearly meant to close the conversation, not open it. It's not wrong, technically. It's just answering a liability question when everyone in the room actually needed an engineering one.

Tess Calder: The certificate is good evidence for a lawyer. It's terrible evidence for a postmortem.

— VI. Why the Gap Isn't Fraud —

Tess Calder: I want to say something in their defense, actually, because I don't think this client lied to anyone.

Marcus Hale: They didn't. That's the part I'd want listeners to sit with. Nobody forged a sign-off. Nobody hid the new onboarding tool from the auditor on purpose, it simply didn't exist yet when the fieldwork happened. The report was correct when it was written. The organisation changed underneath it, the way every organisation changes constantly, and nobody owned the job of noticing that a specific certified control had just grown a new blind spot.

Tess Calder: So the failure isn't dishonesty. It's that nobody's job was "keep the thing the audit certified still true after the audit ends."

Marcus Hale: Which is a genuinely strange gap when you say it out loud. We'll spend six figures and three months proving a control worked for a sampled period, and then spend approximately nothing making sure it keeps working the other three hundred and sixty days we didn't sample.

— VII. The Scope Line Nobody Reads —

Tess Calder: There's a second version of this that isn't even about drift. It's about what was never inside the line to begin with. Give me an example, because this is the one I use most often as an attacker.

Marcus Hale: A manufacturing client, different engagement, scoped their ISO 27001 certification around the corporate IT environment, email, file shares, the HR system, the usual perimeter. Completely reasonable scope, clearly documented, nothing hidden. The operational technology network, the actual factory floor systems, sat outside that boundary by design, because OT security is its own discipline with its own standard and nobody wanted to force it into a framework built for office IT.

Tess Calder: And the factory floor is exactly where I went in, on an engagement for the same client, eight months after they'd framed that ISO certificate in the lobby.

Marcus Hale: Through a vendor remote-access box sitting on the OT network, that nobody outside the plant even knew existed, let alone scoped into anything.

Tess Calder: The certificate wasn't wrong. It was accurate about exactly what it covered. Everyone just quietly stopped remembering there was a line, and started treating the certified part as if it described the whole company.

Marcus Hale: The scope boundary is usually the single most important sentence in the entire report, and it's also the sentence nobody outside the audit team ever actually reads.

— VIII. Tess's Recon —

Tess Calder: I'll tell you exactly how I use that sentence, because it's not subtle. Before I ever touch a client's network, if they have a published SOC 2 or ISO scope statement, I read it first, on purpose, looking for precisely what it excludes.

Marcus Hale: You're using their own compliance paperwork as reconnaissance.

Tess Calder: Every time. A scope statement is a company telling me, in writing, with a lawyer's signature on it, exactly which systems they decided not to hold to the standard. That's not a leak. It's published. Nobody reads it as a target list except the people whose job is to read everything as a target list.

Marcus Hale: So the document meant to build confidence is also, read correctly, a map of where confidence specifically wasn't earned.

Tess Calder: The best fifteen minutes of prep I can spend on any engagement is reading the client's own certificate for what it carefully doesn't say. Half the time the gap between the scoped environment and the real one is sitting right there, in plain language, in a document they handed me themselves.

— IX. Controls That Work on Paper —

Tess Calder: There's a third version too, and this one bothers me more than the other two, because it doesn't even require anything to change. A control can satisfy an auditor and still never have done anything.

Marcus Hale: Say more.

Tess Calder: "We have a documented patch management policy requiring critical patches within thirty days." That sentence satisfies a control. An auditor confirms the policy document exists, interviews someone who describes the process, maybe samples a handful of tickets that closed on time. None of that tells you whether, across the actual fleet, patches are landing within thirty days or within four months, because the auditor sampled tickets, not the fleet.

Marcus Hale: The policy is real. The practice it describes might be real for the five tickets someone happened to pull, and aspirational for the other eleven hundred.

Tess Calder: I've walked straight through a "critical patches within thirty days" shop on a box that had been unpatched for over a year, sitting right next to a policy document that, on paper, made that exact box someone's job to have already fixed.

Marcus Hale: The sampling problem gets worse the bigger the fleet gets, too. A handful of tickets might be a fair sample at fifty servers. At five thousand, across a dozen teams who each interpret "critical" with slightly different judgment, a sample of a dozen tickets tells you almost nothing about the other four thousand nine hundred and eighty-eight.

Tess Calder: So the control doesn't even need anyone to cut a corner on purpose. It just needs to be large enough that sampling stops being representative, which happens quietly, as a side effect of success, the moment the company that passed the audit grows past the size the sample was ever designed for.

— X. The PSIRT Angle —

Marcus Hale: This is also exactly where PSIRT sits in the picture, and it's worth saying plainly since it's live for us right now. A vulnerability- handling process is one of the controls that's easiest to certify on paper and hardest to keep continuously true in practice, because its entire job is reacting to things nobody can schedule in advance.

Tess Calder: An auditor can confirm you have an intake process, a triage process, a disclosure timeline, all documented, all demonstrated on a sample of past reports.

Marcus Hale: None of which guarantees the next report, filed at two in the morning on a holiday weekend, gets handled with the same rigour as the three tickets the auditor happened to sample. A PSIRT that's certified and a PSIRT that's actually staffed, resourced, and current are two different claims, and only one of them is what the report checked.

Tess Calder: So "we have PSIRT" is a snapshot too, unless something keeps re- verifying it's still true.

Marcus Hale: Unless something keeps re-verifying it's still true, continuously, not once a year. Which is the actual argument for a PSIRT being run as a standing, always-on function rather than a project someone stood up for an audit and then quietly stopped staffing. The good version doesn't get re-certified once a year, it proves itself every single time a report comes in, at whatever hour it arrives.

Tess Calder: So the honest answer to "are you compliant on vulnerability handling" isn't a report at all. It's whatever happened the last time a report actually landed at two in the morning.

Marcus Hale: That's the only test I actually trust anymore. Not the documented timeline. The real one, from the last time it got used without warning.

Tess Calder: Is that the actual argument for running PSIRT as a managed, standing service instead of something a company bolts together once a year before an audit? Because that's a harder sell than it sounds, if the pitch is just trust us, it'll be current.

Marcus Hale: It's not trust us. It's watch us, continuously, instead of once a year. A standing PSIRT means the intake process gets exercised every week, not demonstrated for a sample once, which is the whole difference between a control that's current and a control that's merely documented. The clients who've moved to that model stopped dreading the renewal audit, because by the time it shows up, there's nothing to reconstruct. It's just describing what already happened, on schedule, all year.

— XI. What Actually Closes the Gap —

Tess Calder: Fine. What's the version of this that doesn't just mean "audit more often and hope," because that's not a fix, that's a bigger bill for the same snapshot problem.

Marcus Hale: It isn't auditing more often. It's decoupling verification from the audit calendar entirely. The clients who actually close this gap run continuous control monitoring; automated, frequent checks that re-confirm, every day or every week rather than once a year, that the access review actually ran, that the patch SLA is actually being met across the real fleet, not a sample of it. The audit becomes a report on data you already trust, instead of the one moment a year anyone actually looked.

Tess Calder: And the scope line.

Marcus Hale: Gets reviewed on a trigger, not a calendar. Any time a new system, a new vendor connection, a new onboarding tool gets added, someone explicitly asks whether it falls inside or outside the certified boundary, before it quietly becomes the exception nobody scoped.

Tess Calder: So it's the same shape as everything else this show keeps landing on. An inventory, kept current, with an owner, instead of a document produced once and framed in a lobby.

Marcus Hale: I don't think this show is ever going to run out of versions of that sentence either.

— XII. What It Costs —

Tess Calder: What did continuous monitoring actually cost the clients who built it properly, because I don't want this to sound like a free upgrade.

Marcus Hale: Real tooling spend, and a bigger cultural cost than people expect: someone has to own daily or weekly control checks as an actual job, not a once-a-year fire drill the compliance team runs before the auditor shows up. For a mid-size client, that was roughly two additional headcount and a monitoring platform, call it a few hundred thousand a year, ongoing, not a one-time project fee.

Tess Calder: Compared to a breach that happens ten weeks after a clean report.

Marcus Hale: Compared to the actual cost of that breach: incident response, notification, the client relationships that didn't survive it, and a board that no longer trusts the next clean report either, which is its own expensive, lingering problem. Continuous monitoring is the boring, ongoing expense. The alternative is the dramatic, occasional one, and dramatic and occasional is reliably more expensive than boring and constant, once you actually add it up.

Tess Calder: Who actually signs off on two extra headcount for something that, if it works, produces no incident to point to? That's the same problem we keep running into. Prevention doesn't make a good slide.

Marcus Hale: It didn't get approved as a security line item, if I'm honest. It got approved because the sales team wanted to answer "how current is your compliance" with something better than "as of our last audit, eleven months ago," on calls with enterprise prospects who were starting to ask that question directly. Security got the budget. A sales conversation is what actually moved it.

Tess Calder: Same shape as the funding story from a few episodes back. The security argument opens the ticket. Something else pays for it.

Marcus Hale: I've stopped expecting the security argument to win on its own. I've started looking for whichever other argument happens to want the same thing.

Tess Calder: Boring and early again.

Marcus Hale: Boring and continuous, this time. Early was last episode's version of the sentence.

— XIII. Compliant and Secure —

Tess Calder: I want to push back gently on one thing, because this could land as "compliance is theater," and I don't think that's actually our argument.

Marcus Hale: It isn't. I'd take a client with a clean SOC 2 over one without, every time, all else equal. The certificate proves someone did the structured work of defining controls and demonstrating them once, which is real, necessary work most companies never do at all. Compliant isn't the opposite of secure. It's a dated, scoped, sampled claim about security, mistaken for an ongoing one.

Tess Calder: The fix isn't distrusting the certificate. It's knowing exactly what question it already answered, and building something that keeps answering the question it didn't.

Marcus Hale: Snapshot assurance isn't a reason to stop taking the photograph. It's a reason to stop pretending the photograph is a live feed.

Tess Calder: It's also, I think, the cleanest answer we've given yet to the question this whole show keeps circling. Borrowed certainty was a guess wearing a number's clothes. Borrowed severity was someone else's answer wearing yours. This one's the same shape at the scale of an entire company: a Tuesday wearing a year's clothes.

Marcus Hale: Every version of it fixed the same way, too. Not by trusting less. By checking more often than the thing you're trusting was ever built to update.

— Outro —

Jessy Lang: Snapshot assurance: a true claim, dated, scoped, and quietly treated as timeless the moment it gets framed and hung in a lobby. I liked that this one didn't ask anyone to distrust their own audit, just to be honest about which Tuesday it was describing. If you've ever been breached ten weeks after a clean report, or you're the one who built continuous monitoring and watched the next audit get boring, tell us. We read everything. Subscribe wherever you're listening to this, and leave us a rating if you've ever framed a certificate for a boundary nobody else in the building remembered was there. Until then.