In our last post, we wrote that AI doesn't fix the connective tissue problem. It runs into it, faster and more expensively. That was about performance. This is the sharper version, because ungoverned AI doesn't just underperform.
It acts.
It answers a customer. It flags a patient. It scores an application. It shapes a decision that someone then makes and signs. Every one of those is an action taken in your organization's name. And in a lot of organizations right now, no one can say who owns it, who checked it, or how to trace it back. That first part isn't rhetorical. In McKinsey's global survey of roughly 1,500 organizations, 28 percent said their CEO was responsible for overseeing AI governance and 17 percent said their board was. On average, two leaders share it. Shared ownership works fine right up until the day it has to resolve to one name.
That isn't a model problem. It's the same gap we find everywhere execution breaks down: the parts work, and nothing connects them to accountability.
Most of the AI risk conversation happens at the level of the model. Is it accurate, is it biased, could it be tricked. Those are real questions. They're just not where the damage has been showing up.
IBM and Ponemon studied 602 organizations breached between March 2025 and February 2026. Sixty-eight percent had no policy governing AI use or shadow AI. Among those breached through their own AI, 92 percent lacked adequate access controls. Breaches involving a company's own AI models went from 13 percent to 21 percent in a single year, and incidents involving shadow AI, tools employees adopted with no security sign-off, more than doubled to 43 percent. Only four in ten organizations restricted access to their AI systems at all.
Grant Thornton came at it from the other direction this spring, asking 950 executives whether they could pass an independent AI governance audit within 90 days. Seventy-eight percent were not strongly confident they could. Their phrase for the condition is organizations "scaling AI they cannot explain, measure or defend."
The reported failure is almost never that the model was wrong. It's that nobody had decided who could point it at what.
The damage tends to come from four places, and the model isn't any of them.
Decisions no one owns. A model makes a call, or shapes one, and there's no clean answer to who is accountable for the outcome. Not who built the tool. Who owns what it decides.
Access no one reviewed. A team stands up something genuinely useful, and it reaches into systems and records that security, privacy, and legal never signed off on. The tool may already be breaking a policy, a contract, or a regulation. Nobody knows, because nobody was asked.
Outputs no one validated. A recommendation gets acted on at a volume no person is actually checking. When it's right, it's invisible. When it's wrong, it has already happened.
Rules no one connected. The obligation exists, in a policy, a regulation, a contract. The tool was never told. In regulated environments, that gap is not theoretical.
None of these require a bad model. An accurate, unbiased, well-secured model produces every one of them, because none of them are about the model.
Which is not to say the model is never the problem. A biased one, an inaccurate one, a poorly secured one: those are real risks, they're worth managing, and sometimes they are the whole story. The point is that they're additive, not alternative. Fixing the model doesn't touch any of the four above, and most organizations are spending their attention on the half they can benchmark. Exposure scales with what the AI is allowed to do, not just with how well it does it.
Ask a room how AI is governed and the answer is usually that there's a human in the loop.
That the phrase means less than it sounds is not a new observation. Researchers have been making it since Lisanne Bainbridge's "Ironies of Automation" in 1983, and a steady run of writers have made it again in the last two years. Take the diagnosis as settled. The argument worth having is about the cure.
The phrase has always done two jobs at once. It catches errors, and it names someone responsible. At human scale the two held together, because the person accountable was close enough to the work to see the problem.
AI pulls them apart. The error-catching drowns under volume. The naming stays perfectly intact. So you keep the half that reassures you and lose the half that protects you: a named person who takes the fall for output they had no practical way to inspect.
You do want a single owner. One clear “throat to choke” is the right instinct and best practice. But the failure isn't naming the owner. It's naming one who can't see or stop the thing they answer for.
Which is why the reflex fix tends to disappoint. Add a policy. Stand up a committee. Insert an approval step. If you have no governance at all, do this, it beats nothing! But most of it becomes one more place where a person signs off on something they can't really see.
Healthcare ran into this failure mode with automated alerts, before AI. Hospitals wired their systems to raise a safety alert for every possible drug interaction, with a clinician squarely in the loop on every one. The alerts fired so often, on so many things that didn't matter, that clinicians learned to click past nearly all of them. Isaac and colleagues studied 233,537 alerts and found prescribers overriding 90.8 percent of drug-interaction warnings, and dismissing the high-severity ones at 89.6 percent, which is to say at essentially the same rate as the trivial ones. The control was there on paper and dead in practice, and nobody felt the danger, because the box was checked on every screen. What eventually helped was redesigning the alerts: tiering them by severity, hard-stopping the few that were genuinely dangerous, and demoting the rest to informational. Compliance went from 10 percent to 29 percent, and to 100 percent on the most severe tier. Fewer alerts, more oversight.
Governance built as a checkpoint is just a brake. Governance built into how the work actually runs is closer to steering. Someone owns each decision, the system shows its work, and you can reconstruct what happened when something goes wrong.
I want to be careful not to oversell that. Accountability is not what makes an organization fast. The fastest movers are usually the ones with the most money riding on moving, and they will outrun their own governance whether or not anyone designed it. The darker read, that AI will eventually outpace any human capacity to control it, isn't unreasonable either.
What structure changes is narrower, and worth more than it sounds. It decides whether you find out. Whether the speed compounds or ends in a cleanup that costs more than the head start was worth, and whether you learn from your own systems or from someone outside the building.
Which brings me back to a familiar place, at least for WYS. Governance, ownership, and delivery are separate boxes on the org chart and one continuous system in practice. AI didn't create that gap. It goes around the people who were closing it, quietly, with talent and conscientiousness and a good deal of sheer will. That was never free, and it was never going to scale. It just used to be invisible.
For some readers this isn't a thought experiment, it's a filing requirement. Look again at the examples above. Scoring a job application or a line of credit is a high-risk use under Annex III of the EU AI Act. Clinical decision support falls under the medical-device regime. And what the Act asks for in those categories is oddly familiar: Article 14 requires human oversight by someone with the competence and the authority to interrupt the system, and Articles 12 and 19 require automatic logging, retained six months or more, so a decision can be reconstructed afterward. A named owner who can actually stop it, and a record of what it did. Non-compliance in those categories runs to 15 million euros or 3 percent of worldwide turnover. The headline 35 million and 7 percent is a separate tier, reserved for the handful of uses the Act bans outright.
Almost none of that is in force, and less of it than a year ago. The EU's Digital Omnibus, effective 27 July 2026, pushed the Annex III obligations from this month to December 2027, and the product-embedded ones to August 2028. Colorado repealed its AI Act in May, replacing it via SB 26-189 with a narrower disclosure law that drops the duty of care and the impact assessments, effective January 2027. The requirements got lighter and later.
If that reads like a reprieve, read it again. Regulators moved the date because the conformity infrastructure wasn't ready. The exposure is exactly where it was. And the liability that never waited on a statute, a customer harmed, a contract breached, a story with your name in it, was never on that timeline to begin with.
What did change is what it will cost you. Build this on your own schedule and you can design oversight into how the work runs. Build it in the last two quarters before a deadline and you'll get a compliance exercise: a policy, a committee, a sign-off. That is the rubber stamp again, with a due date attached. Oversight with real authority, and logs good enough to reconstruct a decision, are operating-model work. They take time to put in place.
So the question worth sitting with is this: if one of your AI tools made a decision tomorrow that harmed a customer, a patient, or the business, would you know it happened? Would you know who owned it? Could you trace how it was made?
If those answers aren't clear, the problem isn't the AI. It's the system around it.
More on connective tissue in the next post.
Sources
