"The Fix Is Available"

Jessy Lang: Welcome back to Security Dialogues. I'm Jessy, and this is episode three. We've got someone new today... Nora Voss, product lead at a manufacturer of connected industrial equipment, and an EACG client, which makes her the first guest on this show who doesn't work for us and can say whatever she wants about us. Nora, before Marcus and Tess even get a word in... what do you actually do, and why does a product manager end up on a security podcast?

Nora Voss: Thank you for having me, Jessy. And thank you for letting me say some of this out loud... normally the only people who hear it are my own team, behind a closed door. I run a product line that's been shipping for twelve years. Hardware with software inside... the kind that ends up bolted to a wall in a plant somewhere, on a ten-year service contract, sometimes with no reliable network connection at all. Which means I'm not the person who finds vulnerabilities, and I'm not the person who writes the advisory. I'm the person who's still there three years later when somebody asks why a "fixed" device is running the exploited version.

Jessy Lang: That's a very specific kind of job.

Nora Voss: It doesn't have a title. It's just what's left over once everyone else has done their part and gone home.

— I. Who Answers the Question —

Jessy Lang: Give me the moment you realised that was actually the job.

Nora Voss: Two years ago I asked for a simple number: how many units of one particular model are actually out there. Sales gave me a figure. Then I cross-referenced it against warranty registrations and distributor stock movements, and the real number was forty percent higher. Grey-market resale, a second distributor we'd never authorised, units that had changed hands twice before anyone registered them.

Jessy Lang: How do you even find that out?

Nora Voss: By accident, mostly. A customer called in for support on a serial number that, according to my own records, didn't exist. I pulled the thread for a week and found an entire parallel supply chain I had no visibility into. I couldn't patch a fleet I couldn't count. And I run the product.

Jessy Lang: So "how many are out there" isn't even solved before you get to "did they install the fix."

Nora Voss: Correct. And that's before anyone's touched a line of vulnerable firmware.

Jessy Lang: Did you fix the count, at least?

Nora Voss: Partially. I registered the second distributor, backdated as far as the paperwork would allow, and put a serial-number check into our support intake so it can't silently happen again. I still don't believe my current number is the real one. I believe it's closer.

Jessy Lang: All right. Marcus, Tess... this is who's here to talk to you tonight. This week, Nora found out exactly how much that gap costs. Eleven months ago, an advisory closed on the other side of her world... CVSS scored, CSAF published, "fixed" stamped on the ticket. A third of the fleet it was written for never installed it, and one of them just found out why that mattered.

Nora Voss: That fleet was mine. And "fixed" is doing a lot of work in that sentence for a status that, out there, was still very much true.

— II. The Advisory Isn't the Fix —

Marcus Hale: The advisory did what an advisory does. We identified the vulnerability... an authentication bypass in the controller's management interface, remotely reachable if the interface was exposed... developed a fix, tested it, and published it, with a CVSS score and a remediation status marking affected versions as fixed once the patch is applied. That is the job, and it's not a small one. Coordinating that across six product variants, in three languages, on a disclosure timeline, is most of what my team does all year.

Nora Voss: I'm not disputing the work. I'm disputing the word at the end of it. Once the patch is applied. Say that part again, slower.

Marcus Hale: Once the patch is applied.

Nora Voss: That's the sentence that turns your "fixed" into a hope. You didn't fix my fleet. You fixed a version number. Eleven months after you published, eleven months after my customers had every opportunity to know, a third of my installed base was running the vulnerable build... and one of them found that out because somebody used exactly that bypass to sit inside their network for six weeks before anyone noticed. Lateral movement, credential harvesting, the works. By the time their own team caught it, the "fix" had been sitting on my download page for the better part of a year. Your advisory said "fixed" the entire time.

Tess Calder: Six weeks inside a network before anyone noticed... on a vulnerability that already had a name and a patch number. That's not a sophisticated adversary. That's just patience.

Nora Voss: That's the part I can't get past. It wasn't clever. It didn't need to be.

Marcus Hale: The status field describes the vendor's remediation state, not the deployment state. That distinction exists on purpose... otherwise every advisory would need updating continuously, forever, by a party who doesn't control the data.

Nora Voss: I know why it exists on your side of the desk. I'm telling you nobody downstream reads it that carefully. A customer, a regulator, an insurer... they read "fixed" and they file it. Nobody reopens the file to ask whether it actually got installed. You built a precise word for an audience that reads it imprecisely, and you knew that when you chose it.

— III. Why It Doesn't Install —

Tess Calder: Can I ask the obvious question. Why didn't it install?

Nora Voss: Depends which third. Some of it had no connectivity... literally, physically, no path to a fleet-management server. Some of it sat behind a customer's own change-control process, which requires revalidating the whole line before touching firmware, and revalidation costs more than the customer believes the risk does. And some of it needed a technician on-site, because the device doesn't take a remote push... which means a van, a scheduled shutdown, and a service window the customer has to agree to book. One customer refused an update outright, for two years, on a signed statement that revalidating their line would cost more than they were willing to spend against a risk they'd assessed as tolerable. I disagreed with the number. I couldn't disagree with the right to make it... it was their line, their risk register, their signature.

Marcus Hale: Some of that third, though... surely some of it really is just nobody looking.

Nora Voss: Some of it, yes. I'm not here to tell you negligence doesn't exist. I'm here to tell you it's the minority explanation, and it's the only one your process currently has a word for. I sat in a meeting once where someone from a security team said "we'll just push an update," and the room went quiet, because a third of the fleet in question had no telemetry channel to push anything through. He wasn't wrong that a fix existed. He was describing a fleet that doesn't exist... one where every device phones home.

Tess Calder: So it's not that people ignored the advisory.

Nora Voss: Most didn't. Most made a rational decision with the information and the constraints they had, and the information you gave them... "fixed" ... told them there was nothing left to decide.

— IV. The Status That Lied —

Marcus Hale: I want to push back on "lied." The status was accurate about what we controlled.

Nora Voss: Then say what you controlled. You controlled the existence of a fix. You did not control whether it ran anywhere, and you reported both facts using one word.

Marcus Hale: There's a reason that word doesn't carry a percentage. If I attach an installation figure to a status I don't fully trust, I'm putting a number in a regulatory filing that I can't defend if it's challenged. An honest "unknown" was, until tonight, the safer legal position than a wrong number.

Tess Calder: You already publish a number that changes as new facts arrive. The CVSS temporal score moves with exploit-code maturity and remediation level, and nobody treats that as indefensible... it's expected to move. Why would a coverage percentage be any different?

Marcus Hale: Because temporal metrics describe the vulnerability. A coverage number describes my customers, and most of them never agreed to be described in a public document.

Tess Calder: Then anonymise it. Publish the percentage, not the names behind it. You're not protecting the number's accuracy... you're protecting yourself from a conversation with Legal.

Marcus Hale: …That's a sharper version of what Nora just said. Fine... the precedent exists. I don't have a technical excuse left. Only an organisational one.

Nora Voss: It's the safer position for you. It is not the safer position for the sysadmin who reads "fixed" and reprioritises something else instead. You optimised for your own defensibility and called it caution.

Marcus Hale: …That lands too. And it's the same move you made with Tess two weeks ago about the risk matrix. One number standing in for two different things, and everyone downstream assumes it's one.

Tess Calder: Wait. I need to sit with that, because I think I've been doing something adjacent for years and never named it.

— V. Tess's Attack Surface Has Always Included This —

Tess Calder: A good chunk of my initial-access findings on engagements are already-patched CVEs. I always logged that as a patch-management failure... a checkbox on the report, a paragraph about needing better hygiene. I never once asked why the patch wasn't there. I assumed neglect.

Nora Voss: Sometimes it's neglect.

Tess Calder: Sometimes. But there was one engagement... an energy client, a control room I got into through a controller running firmware eighteen months out of date. My report called it a patch-management failure, one line, moved on to the next finding. What I never asked was why an eighteen-month-old vulnerability was still sitting there. Turned out the fix required exactly what Nora's describing: a five-figure revalidation, a shutdown window they couldn't get approved twice in one budget cycle, and a customer who'd done that math and decided the exposure was cheaper than the fix. My report called that a failure. It wasn't. It was a decision, made by someone with a budget, and I never even asked who made it.

Marcus Hale: I read that report. I remember the line. I didn't ask either ... it was a closed finding by the time it crossed my desk, filed under "known issue, customer accepted," and I moved on to the next advisory.

Nora Voss: That's the finding I never get invited to help write. The report just says "patch," full stop, like patching is free, and then it disappears into two different filing cabinets that never talk to each other.

— VI. When Patching Faster Is Wrong —

Marcus Hale: Then let me put the other side on the table, because it isn't only slow. Push too fast and you can do real damage.

Nora Voss: I have the scar for that one. Three years ago, different product line... we shipped a security fix under real pressure, a researcher was about to go public, and we compressed six weeks of validation into two. The update shipped. On a subset of units with an older bootloader revision we hadn't tested against, it bricked them. Not "needs a reboot" bricked... dead, needs-a-truck-roll-to-replace-the-board bricked. What started as a fix became a recall, on a fleet that included medical-adjacent equipment. We traded eleven months of a moderate, contained exposure for a guaranteed, total outage on every unit we touched.

Marcus Hale: Eleven months of moderate, contained exposure. That's the trade I'd have made too.

Tess Calder: Would you. Say the word again... "moderate." Where does that number come from?

Marcus Hale: From the same place it always does. Likelihood, impact, judgement.

Tess Calder: Whose judgement... yours, in an advisory meeting, or Nora's, standing next to the bootloader that's about to become scrap?

Marcus Hale: …Neither of us was there. It's an estimate.

Tess Calder: Then you just did the thing we spent an entire episode taking apart. You put a guess... "moderate exposure"... on one side of a scale, and a fact... a hundred percent of a fleet gets bricked... on the other, and called the guess the safer bet because it looked smaller than the fact. That comparison only means something if both sides are the same kind of number.

Marcus Hale: …And they weren't. Borrowed certainty... my own phrase, coming back around on me. One's a probability nobody measured. The other's a certainty, because Nora had already run it once, on her own fleet.

Tess Calder: So the actual decision wasn't "slow is safer than fast." It was "a known, guaranteed cost against an unknown, unmeasured one"... and we'd call that a bet, not an analysis, if anyone else on this show made it.

Nora Voss: I made exactly that bet. I'd probably make it again. I just didn't know that's what I was doing until just now.

Tess Calder: So the honest version of "why didn't you patch faster" is sometimes "because last time we tried, we destroyed the fleet"... and the honest version of "why didn't you patch slower" is the same sentence with the guess and the fact swapped.

Nora Voss: Sometimes it's cost. Sometimes it's connectivity. And sometimes it's that fast and safe are the same request only if somebody paid for the validation infrastructure to make them the same request. Nobody had.

Marcus Hale: Which means "ship it faster" isn't free advice either. It has the same problem as "fixed"... it sounds like an instruction and it's actually a hope that this time, testing catches everything.

Nora Voss: That's twice tonight I've had a word taken out of my hands and handed back sharper. I'm keeping "the same kind of number"... I suspect I'll be using it on people who've never heard of this show.

— VII. Who Pays to Make It Fast —

Tess Calder: So build the telemetry. Instrument the fleet, get remote push everywhere, and the whole trade-off goes away.

Nora Voss: Say that to Finance and watch what happens. I've asked for that budget three separate years. It gets ranked against a new product line, a warranty-cost reduction, a factory retooling... all of which have a number attached to their return, and mine has "prevents a problem we haven't had yet." I lose that argument every year I don't already have an incident to point at.

Marcus Hale: Until you have the incident, and then it's approved in a week.

Nora Voss: Then it's approved in a week, for the fleet going forward, and does nothing for the units already out there with no upgrade path to add telemetry after the fact. Some of my oldest hardware physically cannot take a connectivity retrofit. The instrumentation gap isn't a funding problem I can solve today. It's a design decision made a decade ago, by people who never imagined this conversation.

Tess Calder: That's not an excuse to stop asking for the budget.

Nora Voss: I never said stop. I said don't act surprised when the answer is still no, and don't put "just add telemetry" in an advisory as if it were a patch.

— VIII. Whose Job Is Installation? —

Marcus Hale: Then whose job is it? We publish, the distributor notifies, the customer installs. Three parties, three responsibilities.

Nora Voss: I found out last year that one of my distributors had never forwarded a single advisory to an end customer. Eighteen months of them. Not maliciously... nobody's job description said "forward the vendor's security advisories," so nobody did it, and I had no way of knowing until an auditor asked to see the distribution logs and there weren't any.

Marcus Hale: So the chain has an unowned link.

Nora Voss: The chain has several. I also carry devices that are twenty years into a plant's operating life, past anything I'd call a normal support window, running on a negotiated end-of-support date I agreed with that customer years ago... which means for some of my fleet, "install the fix" was never even on the table, because the product itself was contractually out of scope for new firmware. That's not a gap in the chain. That's a decision somebody made deliberately, years in advance, and it should be labelled differently from a customer who simply never got the email.

Marcus Hale: Those are two different failures wearing the same word.

Nora Voss: Exactly. And my name is on the product, either way. So when a customer gets breached running a build I fixed fourteen months earlier, I'm the one explaining it... not the distributor who dropped the email, and not whoever signed off on the support window.

Marcus Hale: Do the distributor contracts say anything about this now?

Nora Voss: They do, going forward... advisory forwarding is a term in every agreement I've signed since. It says nothing about the eighteen months before I noticed, and nothing about distributors already under an older contract who never agreed to it at all. I can close the gap for new relationships. I can't retrofit it into old ones without renegotiating every single one.

— IX. The Economics of a Truck Roll —

Nora Voss: And before either of you suggests "just automate the update"... I want the actual arithmetic on the table. A van, a technician, and a scheduled shutdown is not a metaphor. I costed a fix like that last year: technician day rate, travel, the customer's production downtime for the shutdown window, multiplied across a fleet with sites on four continents. The security team wanted it pushed in a month. Finance wanted eleven. We shipped in seven, and I spent the other four defending why it wasn't three.

Marcus Hale: What would have made three possible?

Nora Voss: Telemetry I don't have on half the fleet, remote management I don't have on a third of it, and a customer contract that doesn't currently let me touch their line without a change order. None of that gets fixed by publishing faster.

Tess Calder: So when I write "remediation timeline: unclear" in a report, that number... seven months, not three... is the actual answer, and nobody's telling the customer that's what "unclear" means.

Nora Voss: It's the honest answer. It's just not the one anyone wants typed into a board deck.

— X. The Triage Nora Doesn't Advertise —

Tess Calder: Can I turn this back on you for a second. You've spent this whole conversation describing other people's gaps. Have you ever been the gap?

Nora Voss: …Yes. Last year a pentest... not one of Tess's, a different firm ... came back with a finding that, done properly, was three separate roadmap items. Segment this device class off the main network. Rotate a set of long-lived credentials baked into the firmware at manufacture. Add signed firmware updates so a compromised build server couldn't push something malicious downstream. I had budget and engineering time for one.

Marcus Hale: Which one?

Nora Voss: The credential rotation. Cheapest, fastest, and it closed the finding on paper. I cut the network segmentation because it needed a customer-side change nobody would approve quickly, and I cut signed updates because the effort dwarfed the other two combined. I told myself I'd revisit both within the year. That was fourteen months ago.

Tess Calder: Fourteen months. That number keeps coming up tonight.

Nora Voss: It does. So no... I'm not describing this from outside. I make the same call Marcus makes when he chooses what a status word doesn't say, and the same one Tess's client made when they accepted an eighteen-month-old vulnerability. I just make it with a spreadsheet instead of a form.

— XI. The Regulator Conversation —

Nora Voss: There's a version of this that isn't a customer conversation at all. I once had to sit across from a regulator and explain why a product still inside its official support window had a known, published fix that a meaningful fraction of the fleet had never received.

Tess Calder: What did you tell them?

Nora Voss: The truth, which did not go down well as an answer: that "in support" describes an obligation I owe, not a guarantee about what's running. I could show the advisory, the CVSS score, the date the fix shipped. I could not show them an installation number, because I didn't have one.

Marcus Hale: What would you have shown them, if you'd had it?

Nora Voss: Exactly what we're describing tonight. A percentage, a date it was last updated, and a plan for the rest. Instead I had a document that said "fixed" and a room full of people doing the arithmetic themselves, badly, in front of me. That was the moment I understood this isn't a customer problem or a distributor problem. It's a reporting gap with my name attached, and eventually somebody with actual authority is going to ask the question nobody inside my own company thought to measure.

Marcus Hale: And "we published on time" isn't a defence to that question.

Nora Voss: It's a defence to a different question. Nobody in that room was asking whether I'd published on time.

— XII. What EACG Should Actually Report —

Marcus Hale: All right. Then here's what I take from this, and it changes what I'd put in an advisory. "Fixed" should never be a single word covering two different facts. One fact is: a corrected version exists... tested, dated, available. The other is: what fraction of the known installed base is actually running it. We report the first today and pretend the second doesn't exist, mostly because I don't have that data. Nora does, or her distributors do, or nobody does... and that gap is itself something a customer or a regulator should be told.

Nora Voss: A fleet-coverage number.

Marcus Hale: A fleet-coverage number, reported alongside the fix, updated as it changes, embarrassing when it stays low... because it should be embarrassing. It moves responsibility for closing the gap onto whoever can actually see it, instead of hiding it inside a status word that sounds finished.

Tess Calder: And it gives me something to ask for on an engagement besides "is a patch available." I should be asking for the coverage number before I even start... and for the ones without one, I should assume the worst case Nora just described, not the best one.

Marcus Hale: It also means "fixed" stops being something I get to declare alone. Nora's the one with the number. The advisory should point at hers, not replace it.

Nora Voss: And it means my own triage stops being invisible too. If the coverage number is public, "we cut two of three roadmap items" is a sentence I have to be ready to say out loud, not a spreadsheet nobody outside my team ever sees. I'm not sure I love that. I think it's right anyway.

Marcus Hale: Practically, that's a line in the next advisory template. Not a promise to solve telemetry... a field that says "known deployment coverage: X percent, as of this date, source: vendor estimate," with "unknown" allowed as an honest answer instead of a silent one.

Tess Calder: And a customer or a regulator reading "unknown" would at least know to ask, instead of reading "fixed" and closing the tab.

Nora Voss: "Unknown" would have been a better conversation with my regulator than what I actually had. I'd take that trade today.

— XIII. Nora Concludes —

Nora Voss: I didn't expect to be the one everyone agreed with tonight ... usually I'm the one nobody in the room wants to hear from, because I'm the reason the roadmap has a line item nobody budgeted for. But I'll take it, with one thing added, because I don't want to leave holding only the win. The reason my distributor went eighteen months without forwarding an advisory is that I never asked them to, in writing, with a name attached. I built a disclosure process that assumed the chain would just work, the same way Marcus built a status that assumed "fixed" would just be read correctly, the same way I told myself I'd revisit the two roadmap items I cut. We all published something, or promised something, and hoped the rest would happen on its own. So: Marcus, publish the coverage number, and point to it as the real answer, not a footnote. I'm going back to put a name on every link in that chain, starting with the distributor who's had eighteen months to notice nobody was checking. And I'm reopening the two items I cut, before somebody else finds out the way the regulator found out I didn't have a number.

— Outro —

Jessy Lang: Fleet coverage, not just fix published. I'm putting that in my own section, since apparently slides aren't allowed here. Nora... thank you for coming in and getting agreed with, which I gather doesn't happen to you very often. If you're the one who gets asked "why isn't this patched yet" where you work, tell us how you actually answer it. We read everything. Subscribe wherever you're listening to this, and leave us a rating if a van made you laugh in a security podcast for the first time. Next time, it's back to just the two of them, and Tess is bringing a claim Marcus is going to hate: that most of what gets called a zero-day, isn't. Until then.