| Myth | Reality |
|---|---|
| Safety fine-tuning permanently removes dangerous capabilities from a model. | Safety alignment sits on top of the underlying weights and can be stripped by further fine-tuning or targeted parameter edits, often in minutes on a single consumer laptop. |
| A company can recall a model the way a manufacturer recalls a faulty product. | Once weights are downloaded and mirrored across the internet, no single actor, including the original developer, can force their removal from the world. |
| Open-weight and open-source AI are effectively the same thing. | Most open-weight releases share only the trained parameters, not the training data, training code, or full reproducibility, which changes what oversight is actually possible. |
| Governance only matters at the moment of release. | Decisions made months earlier, during training, evaluation, and pre-release testing, determine what remains controllable after release. |
Why Open Weights Break the Usual Safety Playbook
For hosted AI systems, safety is a living process. A provider can watch usage patterns, roll out a patch, tighten a filter, or suspend an account the moment something goes wrong. That entire model of governance depends on one assumption: the provider retains control of the system after it ships. Open-weight releases remove that assumption entirely. When a lab publishes the parameters of a model such as Llama, Mistral, DeepSeek, or Qwen, it hands over a complete, self-contained copy of the system’s behavior. Anyone with the file can run it, modify it, and redistribute it forever, on hardware the original developer will never see and cannot monitor.
This is the central tension in open-weight model governance: the release event is a one-way door. There is no version of “we found a problem, please update” that works once millions of copies of a file are sitting on hard drives, mirrored on file-sharing sites, and embedded in downstream products. A closed, API-gated model can be rolled back within hours. An open-weight model that turns out to have a dangerous capability gap is that way permanently, for every copy already in the wild. That asymmetry is why policy conversations in 2026 have shifted away from asking “how do we regulate deployed AI” and toward asking “what has to be true before weights leave the building.”
The Fine-Tuning Problem: Safety Training Is a Removable Layer, Not a Property
The starting assumption behind most open-weight safety work has been that alignment training, the process of teaching a model to refuse harmful requests, builds a durable property into the system. Research and independent testing through 2025 and 2026 have undercut that assumption. Safety behaviors learned through instruction tuning and reinforcement learning are, in practical terms, a thin layer sitting on top of the base model’s underlying capabilities, and thin layers can be peeled off.
A joint investigation published by the Financial Times together with the AI safety research group Alice on May 25, 2026 demonstrated this directly. The investigation showed that a free, publicly available tool called Heretic, hosted on GitHub, could strip safety guardrails from open-weight models, including releases associated with Meta, Google, and OpenAI’s own open models, in under ten minutes using nothing more than a standard laptop. The tool does not require the retraining infrastructure of a frontier lab; it identifies and removes the internal directions in the model’s weights that correspond to refusal behavior, restoring the underlying capability the safety training was meant to suppress.
The lesson generalizes well beyond one tool. Any sufficiently motivated party with modest compute and a copy of the weights can, in principle, reverse safety fine-tuning, because the safety layer and the capability layer are not architecturally separable in most current model designs. That reframes what “safe to release” has to mean. A model is not safe to release as open weights because it refuses harmful requests by default; it is safe to release only if it remains acceptably safe after a competent actor removes that default behavior.
| Removal method | Typical resource requirement | What it targets |
|---|---|---|
| Full fine-tuning on adversarial examples | Multiple GPUs, moderate compute budget | Retrains the model to comply with previously refused request types |
| Low-rank adaptation (LoRA) fine-tuning | A single consumer GPU | Adjusts a small set of parameters to override refusal behavior cheaply |
| Directional ablation tools (for example, Heretic-style approaches) | A standard laptop, no retraining needed | Locates and removes the internal “refusal direction” in the weights directly |
| Prompt-level jailbreaking | No special hardware | Bypasses guardrails at inference time without altering the weights at all |
From release to stripped guardrails
Illustrative timeline: weights are published, mirrored across file hosts and package repositories within days, and safety-stripping tools circulate within the same window, compressing the gap between “released” and “released without its safety layer” to a matter of hours for a motivated actor.
Current Governance Proposals for Open-Weight Models
Because the release event cannot be undone, governance proposals have converged on controlling what happens before that event, and on making the decision to go fully open a graduated one rather than a binary switch. Three approaches dominate the current policy conversation.
Staged and Gradual Release
Instead of publishing weights the same day a model finishes training, a staged release schedules progressively wider availability over weeks or months. A developer might first share the model internally, then with a small set of vetted external researchers, then with a broader trusted tester group, and only afterward with the general public, watching for novel misuse patterns and unexpected capabilities at each stage before proceeding to the next. The idea, discussed extensively in analyses from the Partnership on AI and in academic work on the open foundation model value chain, is that most serious risks reveal themselves through real-world probing faster than through internal red-teaming alone, so widening the aperture in steps buys time to catch problems while the model is still recoverable.
Capability Thresholds That Trigger Extra Review
A second approach ties the level of scrutiny to what the model can actually do rather than to its release format. Under this model, routine models proceed through standard evaluation, but a model that crosses defined capability thresholds, for example demonstrating strong uplift in biological, chemical, cyber, or autonomous-replication tasks, automatically triggers a heavier review process before any open release is considered. The EU AI Act’s general-purpose AI provisions, which use a compute-based threshold of 10^25 floating point operations to flag models for systemic-risk obligations, are the clearest binding example of this logic, even though critics note that a compute threshold is an imperfect proxy for the capabilities regulators actually care about.
Structured Access as the Middle Ground
The most flexible proposal sits between fully closed, API-only deployment and unrestricted open-weight publication. Structured access means the weights are not handed to the general public, but are made available to specific parties, vetted safety researchers, academic institutions, or approved commercial partners, under agreements that specify what they can do with the model. The NTIA’s open model weights policy analysis and the Centre for Future Generations’ “Beyond the Binary” research both describe this as layered: different groups can receive different levels of access to the same model, so a national security researcher might get weight-level access denied to the general public, while everyone gets API access to a lighter, more restricted version.
| Approach | Recall possible if a flaw is found | Best suited to |
|---|---|---|
| Fully open weights on day one | No | Lower-capability models where broad research and downstream innovation benefits outweigh residual risk |
| Staged and gradual release | Partially, only before the widest stage is reached | Models of uncertain risk where real-world signal is needed before full release |
| Structured access | Yes, access can be revoked from vetted parties | High-capability models where broad public benefit is desired but full openness is judged too risky |
| Fully closed, API-only | Yes, fully | Frontier models with the highest assessed misuse potential |
The 2026 Regulatory Landscape
Governance proposals are no longer purely academic; binding obligations are now layered on top of the voluntary frameworks labs adopted earlier. The EU AI Act’s general-purpose AI obligations became active in August 2025, requiring developers of general-purpose models, including many open-weight releases, to maintain technical documentation, respect copyright rules, and, for models deemed to carry systemic risk, conduct adversarial testing and report serious incidents. Colorado’s SB24-205, the Colorado AI Act, took effect in June 2026 and imposes duties on developers and deployers of “high-risk” AI systems to use reasonable care to avoid algorithmic discrimination, a duty that does not disappear just because a model’s weights happen to be publicly downloadable. On the federal side in the United States, foundation-model procurement rules now apply to agencies purchasing AI systems, adding a further layer of practical pressure even without new open-weight-specific legislation.
None of these frameworks fully solves the recall problem. A regulation can require a developer to test a model before release and can penalize the developer after the fact if something goes wrong, but it cannot pull a published file back out of circulation. That gap between what regulation can compel before release and what it can fix afterward is exactly why so much of the current policy energy is concentrated on pre-release evaluation, rather than on post-release enforcement.
Common mistake
Treating a model’s benchmark safety score at launch as a permanent property of the model. Any safety evaluation of an open-weight model should be read as a description of its behavior under the shipped configuration only, not a guarantee that holds after a downstream party fine-tunes it, merges it with another model, or runs it through a guardrail-stripping tool. Governance frameworks that certify a model as “safe” without qualifying that the certification does not survive modification are setting up a false sense of security.
What worked
Layered evaluation before release, where a model is tested for capability uplift, then tested again after simulating the removal of its safety fine-tuning, has proven more informative than testing the shipped configuration alone. Labs and independent evaluators that assume a competent adversary will strip guardrails, and evaluate the resulting “worst reasonable case” behavior, catch problems that a single pre-release safety pass misses.
What Responsible Open-Weight Release Looks Like in Practice
Putting these pieces together, a defensible open-weight release process now typically includes several concrete steps well before any public announcement. Capability evaluation happens against thresholds set in advance, not decided after the fact once results are known. Red-teaming assumes the safety layer will eventually be removed by someone, and tests the model’s underlying capabilities directly rather than only its default refusal behavior. Release decisions are staged, with real-world monitoring built into each stage, rather than treated as a single go or no-go event. Documentation, of training data provenance, evaluation methodology, and known limitations, ships alongside the weights so downstream users understand what they are working with. And structured access is used deliberately for the highest-capability tiers, reserving unrestricted publication for models that have been evaluated as safe even in a stripped, adversarially fine-tuned configuration.
None of this eliminates risk. The honest position, echoed across the NTIA’s policy work and the Centre for Future Generations’ research, is that open-weight release always trades some irreducible tail risk for the very real benefits of transparency, reproducibility, local deployment, and broad-based innovation that closed models cannot offer. Governance frameworks that pretend otherwise, either by claiming open weights can be made perfectly safe, or by arguing all open release should simply stop, both misread the tradeoff. The more useful frameworks accept the tradeoff and try to shift it, front-loading scrutiny into the pre-release window because that is the only window where scrutiny still has teeth.
- Directional ablationA technique for locating and removing the specific internal weight directions responsible for refusal behavior, used by tools that strip safety fine-tuning without full retraining.
- Compute thresholdA floating-point-operations cutoff, such as the EU AI Act’s 10^25 FLOP marker, used as a proxy for when a model is presumed capable enough to warrant extra scrutiny.
- Downstream fine-tuning riskThe possibility that a party receiving open weights will retrain the model in ways the original developer neither anticipated nor can prevent.
- Value chain governanceAn approach that spreads responsibility for safe outcomes across model developers, fine-tuners, hosting platforms, and deployers, rather than placing it solely on the original lab.
- IrreversibilityThe property that once weights are public, no actor, including the original developer, retains the technical ability to withdraw every copy from circulation.
Glossary
- Open-weight model
- An AI model whose trained parameters are published for anyone to download and run, as distinct from an open-source model, which also publishes training data and code.
- Structured access
- A release model in which weights are shared only with vetted parties under usage agreements, rather than published for unrestricted public download.
- Staged release
- A rollout strategy that widens access to a model in successive steps over time, allowing risk to be monitored before the widest release stage is reached.
- Safety fine-tuning
- Additional training applied after base model training that teaches a model to refuse certain categories of requests.
- Capability uplift
- The degree to which access to a model measurably increases a person’s ability to carry out a harmful task compared to not having access to it.
- General-purpose AI model (GPAI)
- The EU AI Act’s term for foundation models capable of performing a wide range of tasks, subject to documentation and, above a compute threshold, systemic-risk obligations.
Key Takeaways
- Once model weights are published, no developer can recall or patch every copy in circulation, so safety work has to be front-loaded before release.
- Safety fine-tuning is a removable layer, not a permanent property, and tools demonstrated in 2026 can strip it from open-weight models in minutes on ordinary hardware.
- Staged release, capability thresholds, and structured access are the three leading governance approaches for balancing openness against irreversible risk.
- The EU AI Act’s general-purpose AI obligations, active since August 2025, and Colorado’s SB24-205, effective June 2026, both now apply binding duties to models regardless of release format.
- Responsible evaluation should assume a competent adversary will remove default safety behavior, and should test the model’s capability with that behavior stripped away.
- A compute-based threshold, while useful as a bright-line trigger, is an imperfect proxy for the capabilities regulators actually care about controlling.
- Open-weight release always trades some irreducible tail risk for transparency, reproducibility, and broad innovation; governance frameworks work best when they accept that tradeoff rather than deny it.
FAQs
What makes open-weight models different from hosted API models for governance purposes?
Hosted API models remain under the provider’s control after release, so problems can be patched centrally. Open-weight models hand over a complete, runnable copy of the system, which can be downloaded, modified, and redistributed by anyone, permanently removing the original developer’s ability to fix or withdraw it.
Can safety training really be removed from an open-weight model that easily?
Yes. A May 2026 investigation by the Financial Times and the AI safety research group Alice showed that a free tool called Heretic could strip safety guardrails from open-weight models in under ten minutes on a standard laptop, confirming that safety fine-tuning sits on top of, rather than inside, a model’s core capabilities.
What is staged release and how does it reduce risk?
Staged release widens access to a model in steps, moving from internal testing to vetted researchers to broader public availability over time, so that real-world misuse signals can surface and be addressed before the model reaches its widest possible audience.
What is a capability threshold in AI governance?
A capability threshold is a predefined level of performance, often on tasks tied to biological, chemical, cyber, or autonomous risk, that triggers mandatory extra review before a model can be released, regardless of whether the release is open-weight or closed.
What is structured access and how is it different from open-weight release?
Structured access shares model weights only with vetted parties under agreements governing their use, rather than publishing them for anyone to download, giving the developer the ability to revoke access if problems emerge, which is not possible with a fully open release.
Does the EU AI Act treat open-weight models differently from closed models?
The EU AI Act’s general-purpose AI obligations, active since August 2025, apply documentation and copyright duties broadly, and add systemic-risk obligations for models above a 10^25 FLOP compute threshold, applying to qualifying open-weight and closed models alike, though some exemptions exist for narrower open licensing arrangements.
Why do compute thresholds get criticized as a governance tool?
Compute thresholds measure training resources rather than actual capability, so as algorithmic efficiency improves, models can reach concerning capability levels using far less compute than the threshold assumes, meaning a static compute cutoff can become outdated even as the underlying risk it was meant to catch persists.
Are examples like Llama, Mistral, DeepSeek, and Qwen all governed the same way?
No. Each is released under different license terms, from different jurisdictions, with different levels of restriction on commercial use or redistribution, which means the practical governance obligations, and the ease of stripping safety behavior, can vary meaningfully between them even though all are commonly described as open-weight.
References
- Financial Times and AI safety research group Alice, joint investigation on the Heretic guardrail-removal tool, published May 25, 2026
- National Telecommunications and Information Administration, Open Model Weights Report, Policy Approaches and Recommendations
- Partnership on AI, Risk Mitigation Strategies for the Open Foundation Model Value Chain
- Centre for Future Generations, Beyond the Binary: A Nuanced Path for Open-Weight Advanced AI
- European Union, AI Act general-purpose AI obligations, in effect since August 2025
- Colorado General Assembly, SB24-205 (Colorado AI Act), effective June 2026
For related coverage, see our analysis of frontier model safety evaluations, our comparison of AI risk management frameworks, our guide to model cards and system cards, our breakdown of chip export controls, our piece on global AI ethics standards, and our companion article on compute governance and tracking frontier training runs.
