"You Just... Swapped It?"
Tess brings something she didn't see coming: one of Marcus's clients swapped out their entire encryption scheme, cleanly, in a week, after she flagged a weakness. For once she's not the one with the answers.
AI-narratedShow notes
Shownotes hier ergänzen.
Transcript
Jessy Lang: Welcome back to Security Dialogues. I'm Jessy, and this is episode six. Just the two of them tonight, and for the first time since we've been doing this, Tess is the one walking in without an answer already in her pocket. This week, Tess finished a re-test on an old engagement. Eight months ago she'd flagged a weakness in how a client encrypted data at rest — nothing exotic, just an aging algorithm choice nobody had revisited in a decade. She expected the usual fight. Instead, the client's own report showed the whole scheme had been swapped, cleanly, in about a week, months before she even came back to check.
Tess Calder: I have re-tested maybe forty of these over the years. The average outcome is a ticket that's still open, or a "compensating control" that turns out to be a memo nobody read. I have never once seen "swapped the entire encryption scheme" land in a week with zero drama. I read the report twice assuming I'd misunderstood it.
Marcus Hale: You hadn't. That was one of mine.
Jessy Lang: Marcus, Tess — this is where tonight starts.
— I. The Week Nothing Broke —
Tess Calder: Walk me through it, because I genuinely don't understand how this doesn't turn into an eighteen-month program with a steering committee.
Marcus Hale: Because the client had already done the expensive part years before the finding ever existed. The encryption algorithm wasn't hardcoded anywhere in their application logic. It was a named entry in a small configuration layer that every service already called instead of rolling its own. Changing the algorithm meant changing one entry and running an existing, already-tested rotation path. Nobody touched business logic. Nobody re-architected anything. They tested the new setting in staging for two days, flipped it in production, and watched the dashboards.
Tess Calder: That's it. That's the whole story.
Marcus Hale: That's the whole story. Which is exactly why it never became a story anyone tells. Nothing broke. Nobody worked a weekend. There's no postmortem, because there was nothing to mortem.
Tess Calder: I've spent my whole career assuming a finding like that either gets ignored or becomes a crisis. It never occurred to me there was a third option where it's just... Tuesday.
Marcus Hale: What did you expect, walking in?
Tess Calder: Denial first, usually a risk-acceptance memo signed by someone who'll have moved teams before it matters. Or the opposite: A panic response, an all-hands, a freeze on releases while eleven people argue about which replacement algorithm to use, for months, because nobody had ever had that argument before and everybody wanted to be the one who picked correctly. I've watched both. I've never watched neither.
Marcus Hale: The argument about which algorithm to use took them forty minutes, in a meeting that was already scheduled for something else. They picked the one their configuration layer already supported as a fallback, because someone had added that fallback two years earlier as a matter of habit, not because of your finding.
Tess Calder: So the hardest decision in my entire mental model of this process didn't happen, because someone else had already made it, in advance, for a reason that had nothing to do with me.
— II. What Crypto-Agility Actually Means —
Marcus Hale: The word for the third option is crypto-agility, and I want to be precise about it, because it gets used loosely. It doesn't mean "we use strong encryption." It means: if the algorithm we're using today turns out to be wrong — broken by research, deprecated by standards bodies, outpaced by hardware, whatever the reason; we can replace it without rewriting the system around it. Strength is a property of today's choice. Agility is a property of how easy tomorrow's choice is.
Tess Calder: So you can be agile with weak crypto and rigid with strong crypto.
Marcus Hale: Both happen constantly. Rigid-and-strong is actually the more dangerous of the two, because it feels safe right up until the day the strong algorithm stops being strong, and then you discover you built ten years of assumptions on top of a choice nobody left a door open to change.
Tess Calder: Which is what would have made my finding, on a rigid system, turn into the eighteen-month program.
Marcus Hale: Exactly what would have happened. The vulnerability wasn't the crisis. The absence of a door was always going to be the crisis, whenever it eventually got triggered. By you, by a researcher, by an actual attack. You just happened to be the one who found the door already built.
— III. Why Most Systems Can't Do This —
Tess Calder: So why doesn't every system have the door? This isn't exotic advice. Nobody's surprised to hear "make your crypto swappable."
Marcus Hale: Because building the door costs something today, for a benefit that might not arrive for years, and every roadmap meeting rewards the feature that ships this quarter. I've watched teams hardcode an algorithm name directly into a wire format, bake key lengths into a database schema, scatter cryptographic calls across forty files with no shared layer — not because anyone decided rigidity was fine, but because nobody decided anything. The rigid version is what you get by default when nobody spends the afternoon building the abstraction on purpose.
Tess Calder: And by the time someone like me shows up with a finding, that afternoon is now a much bigger afternoon.
Marcus Hale: It's now the eighteen-month program, because the algorithm choice has had years to grow roots into places nobody remembers to check. I've seen a company discover their encrypted backups from six years ago used a format so specific to one abandoned library that restoring them at all, let alone re-encrypting them, became its own separate project. And I've seen the quieter version, which is almost worse: a company whose crypto was fine, technically swappable in principle, except the only two engineers who understood how it was wired together had both left, and the knowledge left with them. The code could change. Nobody left who could safely change it.
Tess Calder: So agility isn't just an abstraction layer. It's also whether the people who understand it are still in the building.
Marcus Hale: Which is why the inventory matters as much as the code does. An abstraction layer nobody's documented is a locked door with the key buried somewhere in one person's head.
— IV. The Inventory Nobody Has —
Tess Calder: How did your client even know where all their crypto lived, to swap it cleanly? That's the part I can't picture. I do this for a living and I usually can't get a straight answer out of a client about what encrypts what.
Marcus Hale: They'd built what I'd call a cryptographic bill of materials, a year before your finding ever existed. Which algorithm? Which key length? Which library used where? For what?. Not because they predicted your specific finding. Because after a different, unrelated audit, someone got tired of not being able to answer "what do we actually use" in under a week.
Tess Calder: So it's the same shape as the SBOM conversation Priya walked us through. Nobody can secure, or swap, or even discuss a thing they can't list.
Marcus Hale: Same shape, different layer. An SBOM tells you what components you depend on. A crypto inventory tells you what cryptographic decisions are load-bearing inside those components and your own code. Most organisations have neither. The ones who build the second one tend to already have built the first, because it's the same discipline pointed at a narrower question.
Tess Calder: And without the inventory, "swap the algorithm" isn't a week's work. It's a scavenger hunt with security implications if you miss a spot.
Marcus Hale: I lived that scavenger hunt at a different client, two years before this one. No inventory, no shared layer, an algorithm change that should have taken a week took four months, because every time we thought we'd found every call site, a service nobody remembered existed would turn up still using the old one, discovered only when it broke. We shipped the fix in pieces, over four months, with the vulnerable version still live in parts of the estate the entire time. That's the version of this story that doesn't get told on a podcast, because it isn't clean enough to be a good story. It's just slow and mildly humiliating.
Tess Calder: Which is presumably why nobody builds the inventory until after they've lived through exactly that.
Marcus Hale: Well, ... or before, if they're lucky enough to have someone insist on it without needing the scar first.
— V. Harvest Now, Decrypt Later —
Tess Calder: Why now? Encryption's been "good enough" for years. What changed that made your client treat this as worth the afternoon before my finding ever showed up?
Marcus Hale: Harvest now, decrypt later! It's not a new idea, but it's the one that's actually moved budgets this year. An adversary with the resources to do it, for example a well-funded state actor, doesn't need to break your encryption today. They record the traffic today, store it, and wait for the day a sufficiently capable quantum computer can break the algorithm that protected it. For anything with a shelf life longer than "embarrassing for a week", say health records, trade secrets, government communications, anything with a real duration of harm, data encrypted today can still be worth stealing today, purely as a bet on tomorrow.
Tess Calder: So the attack already happened, potentially, and nobody would know for years.
Marcus Hale: That's the specific thing that makes this different from every other "upgrade your crypto eventually" conversation we've had for a decade. Normally the fix can wait until the threat is closer. Here, the theft can happen years before the capability that makes it dangerous even exists, which means "wait until it's closer" is exactly the decision that loses.
Tess Calder: Give me a concrete case where the shelf life actually matters, because "health records" is easy to say and I want to know what it looks like in practice.
Marcus Hale: A pharmaceutical client, different engagement, was encrypting clinical trial data with a lifetime measured in decades — patent timelines, competitor intelligence, regulatory exposure, all of it still damaging fifteen or twenty years out. If someone's been quietly recording that traffic since last year, the theft already happened. The only question left is when decrypting it becomes cheap enough to bother. That's not a hypothetical threat for them. That's a clock that's already running, on data that's already gone, whether or not anyone's noticed yet.
Tess Calder: So for them, "harvest now, decrypt later" isn't a future risk at all. It's a past event they can't confirm and can't undo.
Marcus Hale: Which is exactly why the response isn't "encrypt better going forward" alone. It's "make sure the next algorithm change doesn't take four months," because there will be a next one, and the cost of slow keeps compounding on data that has decades left to matter.
— VI. Not About Quantum Hype —
Tess Calder: I want to push on this, gently, because it's starting to sound like exactly the kind of thing we spent an entire episode taking apart two months ago. Manufactured urgency, dressed up in a scarier vocabulary.
Marcus Hale: It's a fair instinct, and half the vendor booths at every conference this year are exploiting exactly that instinct: "post-quantum" slapped on a product the same way "AI-powered" got slapped on everything the year before. I've had vendors try to sell me a "quantum-safe firewall" that turned out to mean nothing more specific than "we updated one library."
Tess Calder: So how do you tell the real version from the sticker?
Marcus Hale: The sticker version asks you to buy a product. The real version asks you to build a capability — the door, the inventory, the tested rotation path — that pays off regardless of which specific algorithm eventually needs replacing, and regardless of whether the quantum timeline turns out to be five years or twenty. My client didn't buy "post-quantum" anything. They built agility, for an ordinary weakness Tess found using nothing more exotic than a checklist. The quantum case is real, and it's not the only case, and that's exactly how you know it's not just marketing ... the fix helps even if the marketed reason turns out to be overstated.
Tess Calder: That's the distinction I was actually looking for. Not "is the threat real," but "does the fix only make sense if this one specific threat is real." Yours doesn't.
— VII. How the Client Actually Did It —
Tess Calder: Give me the boring mechanical version, because I think that's the part that would actually convince someone with a roadmap to spend the afternoon.
Marcus Hale: Three things, and none of them are clever. First: an abstraction layer — every service calls a shared "encrypt this" function instead of choosing an algorithm inline, so the algorithm is a configuration value, not a decision repeated in forty places. Second: an inventory: the cryptographic bill of materials, kept current, not written once and forgotten. Third - and this is the one people typically skip - a rotation path that gets exercised before it's needed. They didn't invent the swap procedure the week your finding forced their hand. They'd tested it eighteen months earlier, on a completely different algorithm change, for a completely different reason, and it worked then too.
Tess Calder: So the muscle already existed. You weren't asking them to learn a new skill under pressure. You were asking them to do the thing they'd already rehearsed.
Marcus Hale: Precisely that. Rehearsed calm, same as we said about threat modelling. The rotation path is only calm on the day it matters if it was boring on the day it didn't.
— VIII. The Second Lock —
Tess Calder: Wait, back up to the week. You changed one entry in a configuration layer and watched dashboards, but that was encryption at rest, you said. The bytes already sitting on disk under the old algorithm didn't rewrite themselves because you flipped a setting. What actually happened to them?
Marcus Hale: Good catch. That's the part I skipped past too fast. There's a difference between the algorithm you're arguing about and the bytes it's protecting, and the trick that made the week possible lives in that gap. Their data wasn't encrypted directly with one long-lived algorithm choice. Each record had its own data key, and that data key was the thing wrapped, encrypted a second time, by a master key using the algorithm we were actually replacing. Swap the wrapping algorithm, and you only have to re-wrap the keys. The bulk data underneath, encrypted with its own key, never gets touched.
Tess Calder: So there are two locks. The outer one, on the key, is what you actually swapped. The inner one, on the data itself, stayed exactly as it was.
Marcus Hale: Two locks, exactly. Envelope encryption, if you want the proper term for it. It's why "swap the algorithm" for them meant re-wrapping a few million small keys, not re-encrypting however many terabytes of records those keys protect. Keys are small. Re-wrapping one takes milliseconds. That's the entire trick behind the week.
Tess Calder: But if the wrapping algorithm gets broken, isn't the outer lock just open? Why does it matter how many times something was wrapped if the lock protecting it doesn't hold anymore?
Marcus Hale: Because the two locks aren't the same kind of lock, and that's the part I glossed over. The data itself is protected by a symmetric algorithm, AES almost always. The key protecting that data is wrapped with an asymmetric algorithm, usually RSA or something built on elliptic curves. Those are two completely different mathematical problems, and a quantum computer doesn't treat them the same way. Shor's algorithm is the one that actually breaks RSA and elliptic curves. It turns factoring a huge number, or solving a discrete logarithm, from impossible into solvable. Grover's algorithm is the one that touches symmetric ciphers like AES, and all it does is roughly halve the effective key length. AES-256 under Grover is still about as strong as AES-128 is today, which is to say, still fine. So when people say quantum breaks encryption, what they usually mean, whether they know it or not, is that it breaks the asymmetric layer. The symmetric layer barely notices.
Tess Calder: So the outer lock is the one actually built out of the kind of math a quantum computer is good at. The inner lock isn't.
Marcus Hale: That's exactly it. Which is also why swapping only the outer lock isn't a shortcut. It's the whole fix. There's nothing wrong with the inner lock that needs fixing.
Tess Calder: Then why use the vulnerable kind of lock at all? If asymmetric is the one with the actual exposure, why not wrap the data key with another symmetric key and skip the problem entirely?
Marcus Hale: Because the reason you reach for asymmetric in the first place has nothing to do with strength. It's access control. A symmetric key can encrypt and decrypt. Whoever holds it can do both. If forty services all need to be able to encrypt new data keys, and you hand all forty of them the same symmetric wrapping key, you've also just handed all forty of them the ability to decrypt every key any of the others ever wrapped. With asymmetric, you hand those forty services the public key. They can wrap all day. Only the key management system, holding the one private key, can ever unwrap anything. You get real separation between who can create and who can read, which a shared symmetric secret can't give you without a lot of extra machinery.
Tess Calder: So the exposure isn't a mistake. It's the price of the access control you actually wanted.
Marcus Hale: It's a trade, made on purpose, by people who understood it at the time. It only turns into a problem decades later, when the math underneath one side of that trade stops holding. Which is exactly why the fix isn't stop using asymmetric. It's make sure you can replace the specific asymmetric algorithm without redoing the access control model you built it for.
Tess Calder: And the finding I raised eight months ago. That was the outer lock. The wrapping algorithm.
Marcus Hale: That was the outer lock. If it had been the inner one, the algorithm actually encrypting the records, I wouldn't be telling you a one-week story.
Tess Calder: Tell me the version where it's the inner one, then. Because I want to know what I'm actually asking a client to sign up for when my finding lands on the wrong lock.
Marcus Hale: An insurance client, different engagement again. Forty-some terabytes of policy records, spread across three regions and two backup tiers, all encrypted directly with the algorithm that needed replacing. No envelope, no second key to hide behind. We had to read every record, decrypt it, re-encrypt it, and write it back, without ever taking the system offline and without ever leaving a window where a crash could lose data under either key. That meant running both algorithms side by side for months, tagging every record with which one currently protected it, and checking that tag on every read. It finished. It took the better part of a year, and it was easily the least glamorous project I've ever run.
Tess Calder: So the inventory has to know which lock, not just which algorithm.
Marcus Hale: That's exactly what it has to know. For every dataset: is this algorithm protecting a key, or protecting the data itself? Because "swappable" means something completely different depending on the answer, and a checklist that just says "AES-256, yes or no" without recording which lock it's sitting on will lie to you about how hard the fix actually is.
Tess Calder: Here's what worries me about the outer lock, though, even with the two-lock trick. If someone harvested a stolen backup, or intercepted traffic, years before your client swapped the wrapping algorithm, don't they already have the old, wrapped key sitting right there next to the data it unlocks?
Marcus Hale: They do. That's the sharper version of "harvest now, decrypt later" for data at rest specifically. If an attacker got a copy of an envelope, the wrapped key plus the data it protects, before the wrapping algorithm was swapped, then whenever that algorithm eventually breaks, they don't need anything else. They already hold both locks. Swapping the algorithm today protects every envelope created today onward. It does nothing for a copy someone already walked out the door with.
Tess Calder: So the two-lock trick makes the swap cheap. It doesn't make the swap retroactive.
Marcus Hale: Nothing makes anything retroactive. That's the actual reason "why now" matters more than people give it credit for. Every day spent on the old wrapping algorithm is another day's worth of stolen envelopes that no future swap will ever be able to protect.
— IX. The CRA Angle —
Tess Calder: Is any of this actually required now, or is it still just good practice that happens to pay off?
Marcus Hale: It's moving toward required, which is part of why I've had this conversation with three other clients this year who weren't thinking about quantum at all. The Cyber Resilience Act expects manufacturers to handle vulnerabilities, including cryptographic ones, throughout a product's support life, not just at launch. A product that ships with an algorithm baked in so deep it can't be updated without a full re-certification isn't meeting that expectation just because the algorithm was strong on day one. Regulators are starting to ask not "what did you ship" but "what can you still change."
Tess Calder: Which turns crypto-agility from an engineering nicety into something closer to what an SBOM became under the same regulation: not optional, eventually.
Marcus Hale: That's exactly the trajectory. And same as the SBOM conversation: the company that builds it because a regulator will ask is doing it too late and too resentfully to do it well. The company that already has it just answers the question.
— X. What It Costs to Build This In Advance —
Tess Calder: I don't want this to sound free, because nothing we've discussed on this show has ever actually been free. What did the door cost, before anyone knew they'd need it?
Marcus Hale: Real engineering time, spent on something that shipped no visible feature. The abstraction layer took a small team about six weeks. The inventory took an ongoing habit, not a project — someone updates it as part of every change that touches encryption, which is a process cost more than a one-time one. The rehearsed rotation cost a weekend, once, and another afternoon each time they practiced it since.
Tess Calder: Compared to?
Marcus Hale: Compared to the client I mentioned earlier, the six-year-old backups nobody could restore — that recovery effort ran into months, cost more than the original abstraction layer would have across its entire lifetime, and still left them with a system that was just as rigid afterward, because they fixed the emergency and not the underlying inflexibility. Boring and early beats expensive and late in basically every version of this story I've watched.
Tess Calder: Who actually signs off on six weeks of engineering time for something with no feature to demo? That's the part I'd expect to die in a planning meeting.
Marcus Hale: It nearly did, the first time this client tried. What got it approved wasn't a security argument at all — it was an engineer pointing out that the same abstraction layer would also make it trivial to support a customer who had a contractual requirement for a specific algorithm the company didn't currently offer. Security bought the ticket. A sales requirement paid for it. I've stopped being precious about which argument wins, as long as the door gets built.
Tess Calder: So the pitch that actually works isn't "this protects us someday." It's "this is useful in more situations than the one you're currently worried about."
Marcus Hale: Every time I've seen it get funded, that's the version that worked. The purely defensive pitch competes with every other roadmap item for the same scarce urgency. The flexibility pitch competes for nothing — it's just capability, sitting there, waiting to be useful for whatever comes up first.
— XI. Tess Asks the Attacker's Question —
Tess Calder: Let me bring something back to this that's actually mine, because I don't want to just sit here agreeing for twelve chapters. From where I sit: does any of this make my job harder? Does crypto-agility, done well, actually change what an attacker like me can do?
Marcus Hale: Does it?
Tess Calder: Less than you'd hope, and I think that's worth saying honestly. Agility doesn't stop me from finding an implementation bug — a bad random number generator, a key stored somewhere it shouldn't be, a side channel. None of that cares whether the algorithm is swappable. What it does change is the conversation after I find something. On a rigid system, a weak algorithm is a permanent finding that sits on every report for years, because nobody can afford to fix it. On an agile one, it's a finding with a fix attached, and it stops showing up on the next report. It doesn't shrink my attack surface today. It shrinks the list of things everyone's quietly given up on defending.
Marcus Hale: Which is its own kind of honest metric. Not "did agility stop an attack." Did it stop a known weakness from just sitting there, tolerated, because fixing it was unthinkable.
Tess Calder: That's the version I believe. I'd have been skeptical of "this stops attackers." I'm not skeptical of "this stops permanent findings."
Marcus Hale: Say more about "permanent findings," because I don't think that phrase exists anywhere in how I normally talk about this, and I think it should.
Tess Calder: Every red team report has a section of findings that just get re-listed, engagement after engagement, year after year, because the fix was always technically possible and always practically unthinkable. Those findings stop meaning anything. Everyone in the room has learned to read past them. The dangerous version of "assumed permanence" isn't that nobody knows about the weak algorithm. It's that everybody knows, and it's been on the list so long it's become wallpaper.
Marcus Hale: Which means the actual value of what my client built wasn't speed. It was making one specific piece of wallpaper removable again.
— XII. Assumed Permanence —
Marcus Hale: Then let's name the actual failure mode, because I don't think it's "weak crypto," and I don't think it's "not ready for quantum." Call it assumed permanence ... the quiet decision, made by nobody in particular, that today's algorithm is a fact of the system rather than a choice inside it. Nobody sits down and decides "we will never be able to change this." It just becomes true, by default, the day nobody builds the door.
Tess Calder: And the fix was never "predict the future correctly." Nobody on your client's team guessed I'd show up with that specific finding.
Marcus Hale: Nobody guessed anything. They just refused to let this particular choice harden into something unchangeable, the same way they'd refuse to let any other design decision go unexamined for a decade. The quantum case, the compliance case, your finding: three completely different reasons to open the same door. It only had to be built once.
Tess Calder: I came in today just trying to figure out how this didn't turn into the usual mess. I'm leaving thinking that's the actual point. The system that doesn't need me to have the dramatic answer is the one that was built correctly in the first place.
Marcus Hale: Rehearsed calm again. You just found out what it looks like from the outside instead of the inside for once.
— Outro —
Jessy Lang: Assumed permanence — the thing nobody decided, that quietly became true anyway. I like that this episode's villain isn't a bad actor or a bad algorithm. It's a door nobody got around to building. If you've ever inherited a system where "just swap the algorithm" turned into a program with a name and a budget, or you've built the boring version and had it pay off on a random Tuesday, tell us. We read everything. Subscribe wherever you're listening to this, and leave us a rating if a six-year-old backup has ever ruined a week of your life. Next time, we're going back to a thread from a few episodes ago: what happens when the thing generating your "urgent" reports isn't a person chasing a bounty at all, but a tool nobody on your team actually understands, quietly making decisions about what counts as a risk. Until then.