The AI Center of Excellence: Does It Still Make Sense

My Position: The CoE Should Shrink to Governance, Not Disappear Entirely

The AI Center of Excellence was the right answer to a 2023 problem. Enterprises had scattered pilots, no shared vocabulary for risk, and almost no institutional muscle for evaluating a model, a vendor, or a use case. Centralizing that judgment in one function made sense, because almost nobody outside of it had the skill or the mandate to make good AI decisions.

That problem is mostly solved now. AI tooling has commoditized fast: most major software vendors ship a competent AI feature by default, prompt literacy is now closer to spreadsheet literacy than to a specialized data science skill, and a meaningful share of the workforce has used a generative AI tool in their personal life before ever encountering one at work. Against that backdrop, a centralized CoE that insists on being the sole builder of every AI use case has become the bottleneck it was originally created to remove.

My argument is not that governance, shared infrastructure, and cross-team standards should disappear. Those functions are more necessary than ever, precisely because more people across the business are now building with AI independently. My argument is that the CoE’s role should shift from being the builder of record to being the referee, the toolmaker, and the standards body, while the actual building moves into the teams who understand the problem best.

I take this position having watched both extremes fail in practice. Fully centralized CoEs that insist on remaining the sole builder consistently become the reason business units start quietly using unsanctioned tools, because waiting three months for a central team’s roadmap slot is not a realistic option when a competitor ships the same capability in three weeks. But I have also watched organizations swing too far the other way, eliminating central coordination entirely in a burst of enthusiasm for team autonomy, only to discover a year later that nobody can answer a basic question about which teams are using which models on which data. Neither extreme survives contact with an actual audit, an actual regulator inquiry, or an actual competitive pressure to move fast. The federated middle ground is not a compromise for its own sake; it is the only version of this operating model that has held up under both kinds of pressure at once.

The Case for Keeping a Strong Central Function

Governance doesn’t scale by accident

Model risk, data privacy exposure, and regulatory obligations do not become someone else’s problem just because a team outside the CoE built the use case. Without a central body setting minimum standards for evaluation, logging, and human oversight, a fully decentralized organization ends up with as many risk postures as it has teams, and the weakest one becomes the organization’s actual risk profile.

Shared infrastructure is genuinely more efficient centralized

Negotiating enterprise model contracts, maintaining a shared evaluation harness, and operating a common data and prompt-logging layer are exactly the kind of capital-intensive, specialized work that benefits from being built once and reused everywhere. A marketing team and a finance team both need reliable access to models, observability, and cost controls; neither needs to reinvent that infrastructure independently.

Without any coordination, AI still shows up everywhere, just badly

The strongest counterargument to full decentralization is not hypothetical. Organizations that eliminate central AI functions entirely do not see AI usage decline; they see it fragment into inconsistent, ungoverned pockets, often indistinguishable from the shadow AI problem this same cluster covers elsewhere. The absence of a center does not remove risk, it just removes visibility into where the risk lives. Leaders who champion full decentralization as a pure speed play should be honest that they are trading a visible, addressable bottleneck for an invisible, much harder to address one.

Why the Pure Decentralized Model Still Falls Short

Proponents of fully embedding AI talent inside every team, with no meaningful central function, point fairly to speed: a team that owns its own AI capability doesn’t wait in a queue behind other departments’ priorities, and morale is generally higher when a team can act on its own AI ideas without lobbying a central committee. This is a real advantage, and it is the honest reason the fully centralized model of 2023 is dying.

But “no center at all” trades one failure mode for another. Duplicated effort across teams building near-identical capabilities independently, inconsistent data handling that a regulator will eventually notice, and a complete loss of enterprise-wide visibility into where AI is actually making decisions are not hypothetical risks; they are the predictable result of removing coordination entirely. The debate as usually framed, centralize or decentralize, is a false binary. The real question is which specific functions belong at the center and which belong in the team, and getting that allocation wrong in either direction is costly, whether that cost shows up as a slow-moving bottleneck frustrating every business unit or as an ungoverned sprawl of inconsistent AI use that nobody in the organization can fully account for when it eventually matters.

Reading the Maturity Curve Correctly

The right model is not fixed; it depends on where an organization sits on its own AI maturity curve, and the mistake many leadership teams make is applying the model appropriate for their maturity level two years ago. An organization just beginning its AI journey, with little internal AI literacy and no shared risk vocabulary, genuinely benefits from a strong, centralized CoE that builds most use cases directly and trains the rest of the business by example. An organization with meaningfully higher AI fluency, where most functional teams already include someone comfortable working with AI tools, is better served by a federated model where the center governs and equips rather than builds.

The signal that it’s time to shift is not a calendar date, it’s a capability threshold: once a majority of business units have at least one person capable of building and evaluating a basic AI-assisted workflow without central hand-holding, continuing to route all build work through the center becomes a drag on velocity rather than a safeguard against risk.

Leaders sometimes resist this framing because it implies the CoE’s original design was only ever meant to be temporary, and admitting that feels like admitting a past decision was wrong. It wasn’t wrong; it was correct for a moment in time that has since passed. Treating the operating model as something that should evolve on a predictable capability signal, rather than as a permanent structure to be defended, is the mindset shift that separates organizations still arguing about centralization in the abstract from organizations that have already moved on to arguing about something more useful, like which specific standards the federated model should enforce.

• Commoditized toolingMost AI capability now ships inside existing software, reducing the need for a central team to build from scratch.
• Rising AI literacyBusiness teams increasingly include people who can build and evaluate simple AI workflows independently.
• Governance debtFully decentralizing without keeping shared standards creates inconsistent risk postures across teams.
• Platform leverageA shared evaluation and logging layer is cheaper and more reliable built once centrally than rebuilt by every team.
• Federated operating modelThe practical middle ground: central governance and tooling, distributed build and prioritization.

What the Federated Model Actually Looks Like Day to Day

In practice, the federated model that this piece argues for looks less like an org chart redesign and more like a change in who has authority to say yes. A marketing team wanting to build an AI-assisted content workflow no longer files a ticket with a central team and waits weeks for prioritization; they use an internally approved platform, follow published evaluation and logging standards, and build the workflow themselves, with an embedded AI-literate team member who understands the domain far better than a central generalist would. The central function’s involvement shows up before the fact, in the standards and platform the team is required to use, not after the fact, in a queue the team has to wait behind.

This changes what “success” looks like for the central function itself. A CoE measured on how many use cases it personally shipped has an incentive to stay the bottleneck. A CoE measured on how many teams are safely and independently shipping AI use cases on top of its platform has the right incentive: make itself less necessary as a builder, more necessary as a foundation.

The Talent Argument Nobody Wants to Say Out Loud

Part of the resistance to shrinking a centralized CoE is not architectural, it’s personal. A large, centralized AI function is a visible source of headcount, budget, and executive attention, and the people who built that function have a natural incentive to defend its scope even after the argument for keeping it that large has weakened. This is worth naming plainly, because it explains why some organizations keep routing routine, low-risk AI use cases through an overloaded central team long after embedded teams could have handled them faster and just as safely.

The honest counter to this incentive problem is not to eliminate the center, which would throw away real governance and platform value, but to redefine what the center is accountable for so that its incentives point the same direction as the organization’s. A center whose budget and headcount are tied to platform reliability, incident rates, and the number of teams successfully self-serving on its infrastructure has no reason to hoard build work it no longer needs to do.

A Practical Transition Plan for Shrinking the CoE

Organizations sitting on an overgrown, fully centralized CoE rarely benefit from dismantling it overnight; the transition itself needs the same discipline as any other operating model change. A sensible first step is auditing the current backlog and sorting it by risk and domain specificity, since the highest-risk, cross-cutting items are exactly what should stay central, while the domain-specific, lower-risk items are strong candidates to hand to an embedded owner first.

The second step is identifying and training embedded AI leads within business units before removing any central capacity, so there is always a receiving function ready before build responsibility moves. Removing central capacity before the business units are ready simply recreates the fragmentation risk this piece warns against, just under a decentralization banner instead of a centralization one.

The third step is publishing the standards, evaluation harness, and logging requirements the embedded teams will be required to use before they start building independently, so that decentralized speed doesn’t come at the cost of decentralized risk. Skipping this step is the single most common reason a well-intentioned shift toward federation ends up looking, eighteen months later, exactly like the fragmented mess critics of decentralization warned about.

AI Center of Excellence

A dedicated function, historically centralized, responsible for AI strategy, governance, tooling, and often direct delivery of AI use cases across an organization.

Federated operating model

An organizational structure in which a central function owns governance, platform, and standards, while build and prioritization authority sits with embedded teams.

Embedded AI talent

AI-literate staff who sit within a business function rather than a central team, building and evaluating use cases specific to that function’s domain.

Shadow AI

Unsanctioned AI tool use that occurs outside official channels, often a symptom of a central function that has become too slow or too restrictive.

AI maturity curve

A framework describing an organization’s progression from low AI literacy and centralized control toward higher literacy and more distributed capability.

FAQs

Does an AI Center of Excellence still make sense in 2026?

Yes, but its scope should be narrower than it was in 2023. Governance, platform infrastructure, and enterprise-wide standards still benefit from central ownership, while day-to-day building and prioritization increasingly work better when embedded in business teams with rising AI fluency.

What is the biggest risk of fully decentralizing AI without any central function?

The biggest risk is fragmented, inconsistently governed AI use across the organization, where duplicated effort and uneven risk postures replace visible central bottlenecks with invisible, distributed risk that is much harder to audit or correct.

How do you know when it’s time to shift from a centralized to a federated model?

The signal is a capability threshold rather than a fixed timeline: once a majority of business units have at least one person able to build and evaluate a basic AI workflow independently, continuing to route all build work through a central team slows the organization down more than it protects it.

What functions should always stay centralized regardless of AI maturity?

Model and vendor contracting, risk and compliance standards, and shared evaluation or logging infrastructure consistently benefit from central ownership because they are capital-intensive, specialized, and create real duplication risk if rebuilt independently by every team.

Is shadow AI evidence that a Center of Excellence has failed?

It is often evidence that the center has become too slow or too restrictive relative to what business teams actually need, making shadow AI a useful diagnostic signal about where the official structure has fallen behind real demand, rather than simply a compliance violation to shut down.

How should a CoE’s success be measured in a federated model?

Success should be measured by how many teams are safely and independently shipping AI use cases on the CoE’s shared platform and standards, not by how many use cases the CoE itself built, since the latter metric rewards staying a bottleneck.

Does embedding AI talent in business teams increase compliance risk?

It can, if done without shared standards, logging, and a lightweight review mechanism for higher-risk use cases. Paired with strong central governance and platform requirements, embedded teams have not shown materially higher incident rates than centralized ones in organizations that have made this shift deliberately.

What is the most common mistake organizations make in this debate?

The most common mistake is treating centralization and decentralization as an all-or-nothing choice, when the more durable answer allocates specific functions, governance and platform centrally, build and prioritization to embedded teams, rather than moving the entire organization to one extreme or the other.

This debate connects directly to how AI teams are structured more broadly; see our companion piece on restructuring teams around AI. Many of the same trade-offs apply to specialized platform functions, covered in building an ML platform team. Whether a federated CoE model needs an executive sponsor is closely tied to the case for a chief AI officer role. Operational questions around running distributed AI systems safely are covered in agent operations. Getting any of these models to actually stick with staff requires the tactics covered in change management for AI rollouts.

• Tredence, “AI Center of Excellence (CoE): Blueprint to Scale AI Adoption and ROI in 2026”
• Automation Anywhere, “AI Center of Excellence (CoE): The Definitive Guide for 2026”
• Sebastian Straube, “The Strategic Role of AI Centers of Excellence,” Elevate Tech (Medium, 2026)
• Appinventiv, “AI Center of Excellence for Enterprises: Framework, Structure and ROI Guides”
• Rubixe, “What is the AI Centre of Excellence?”