AI risks worth taking seriously—and the safeguards, evidence, and choices that could keep them from defining our future.
The same capabilities that create possibilities can create problems. Explore serious concerns, what could reduce the harm, and what the evidence does—and does not—justify. These are scenarios to manage, not predictions that the worst will happen.
Concern without fatalism
These are things to prevent, not a forecast of inevitable disaster. Each pairs a concrete concern with practical responses and an assessment of what is known. The order is editorial: consequential and thought-provoking first, not a numerical ranking of probability.
Observed harm: relevant AI-enabled harm has already occurred; more extreme versions may not have.
Plausible escalation: a credible extension of demonstrated capabilities, with important barriers remaining.
Uncertain frontier: a serious possibility whose likelihood or prerequisites remain deeply uncertain.
Labels describe evidence, not severity. Uncertainty is neither proof of safety nor proof of catastrophe.
What could go wrong—and what could help?
12 worries · Consequential and thought-provoking first
Tools that accelerate beneficial biology could also lower some barriers for people seeking to cause severe harm.
What could help
Test dangerous capabilities before release: Specialist, controlled evaluations should inform access restrictions and deployment decisions for tools with substantial misuse potential.
How to judge the risk
Serious enough to justify precautions, but not a quantified prediction of catastrophe. Biological knowledge is only part of the problem: physical resources, practical expertise, and successful execution also matter. Those barriers provide some protection; their strength is uncertain and may change.
Tools that accelerate beneficial biology could also lower some barriers for people seeking to cause severe harm.
What could go wrong
As AI becomes more useful for scientific research, some of the same capabilities can be misused. The concern is not that a chatbot spontaneously creates a pandemic, but that better assistance might expand what a malicious actor can attempt.
The 2026 International AI Safety Report describes increasing relevant capabilities and substantial uncertainty about real-world risk. Knowledge benchmarks are not demonstrations of a successful attack; nevertheless, some developers have introduced stronger safeguards because they could not rule out meaningful assistance to novices.
What this could look like
A misuse attempt passes through several weakly connected institutions without any one organization recognizing the overall danger. This is a high-level risk scenario, not a description of an attack method.
What could help
Test dangerous capabilities before release Specialist, controlled evaluations should inform access restrictions and deployment decisions for tools with substantial misuse potential.
Use multiple layers of oversight Responsible access controls, screening by relevant suppliers, institutional biosafety review, and lawful incident coordination can reinforce one another.
Invest in public-health resilience Better detection, diagnostics, response capacity, and medical countermeasures can reduce harm even when prevention fails.
How to judge the risk
Serious enough to justify precautions, but not a quantified prediction of catastrophe. Biological knowledge is only part of the problem: physical resources, practical expertise, and successful execution also matter. Those barriers provide some protection; their strength is uncertain and may change.
Model refusals are not a complete solution, and beneficial research must remain possible. No individual safeguard establishes zero risk.
Signals worth watching
Independent evidence about real-world capability increases, not just higher exam scores.
The effectiveness of safeguards and public-health readiness as scientific tools improve.
Explore the evidence
Sources inform the assessment; they are not predictions that every scenario will happen.
Many coordinated agents could multiply fraud, harassment, cyber abuse, or harmful physical actions faster than people can respond.
What could help
Limit consequential permissions: Restrict what agents can spend, contact, control, or delegate, with approval gates for high-impact actions.
How to judge the risk
Coordinated online abuse already exists, and AI could amplify it. A broadly capable, unstoppable autonomous swarm is not established by present multi-agent demonstrations. Complexity and dependence on infrastructure can constrain attackers, but do not justify dismissing the risk.
Many coordinated agents could multiply fraud, harassment, cyber abuse, or harmful physical actions faster than people can respond.
What could go wrong
A swarm need not be conscious or independently evil. A malicious operator could coordinate many ordinary agents, combining their actions and adapting to feedback. More autonomous systems could increase the scale and speed of this familiar coordination problem.
The more extreme scenario is a resilient, largely autonomous network that keeps pursuing harmful activity despite attempts to stop it. That is a substantially stronger claim than showing that several agents can complete a bounded task together.
What this could look like
A coordinated network of automated accounts targets many organizations at once with deceptive messages and impersonation. A more speculative extension would connect autonomous decision-making to devices capable of physical harm.
What could help
Limit consequential permissions Restrict what agents can spend, contact, control, or delegate, with approval gates for high-impact actions.
Make coordination detectable Platforms and infrastructure providers can monitor coordinated abuse, share appropriately scoped incident signals, and revoke compromised access.
Keep containment independent Network boundaries, hardware interlocks where relevant, and external shutdown mechanisms should not depend only on an agent choosing to cooperate.
How to judge the risk
Coordinated online abuse already exists, and AI could amplify it. A broadly capable, unstoppable autonomous swarm is not established by present multi-agent demonstrations. Complexity and dependence on infrastructure can constrain attackers, but do not justify dismissing the risk.
Restrictions by one provider do not cover every system. Defensive monitoring also needs privacy protections and safeguards against arbitrary exclusion.
Signals worth watching
How independently agents can sustain complex activity outside controlled tests.
Whether defenders can identify, contain, and recover from coordinated abuse in realistic exercises.
Explore the evidence
Sources inform the assessment; they are not predictions that every scenario will happen.
An unreliable system connected to important services could turn a local error into a cascading failure.
What could help
Separate advice from authority: Begin with decision support, staged rollout, and strict limits before granting operational control.
How to judge the risk
AI errors are well established; a particular society-scale accident is not inevitable. The strongest reassurance is demonstrable containment: a system can be wrong without being allowed to cause a catastrophe. Calling the model 'highly accurate' is not enough.
An unreliable system connected to important services could turn a local error into a cascading failure.
What could go wrong
The danger comes from coupling imperfect decisions to consequential actions. A model can misread a situation, rely on incorrect data, or behave differently outside its tested environment. Interconnected systems can amplify an error before a person has time to intervene.
Scale matters. An unreliable suggestion in a draft document is not equivalent to an automated command affecting transport, industrial equipment, financial systems, or an essential service. Shared dependencies can also cause many organizations to fail in the same way.
What this could look like
Several operators rely on the same automated planning service. An unexpected condition produces similar bad recommendations across the network, while rapid automation leaves too little time for local operators to recognize the pattern.
What could help
Separate advice from authority Begin with decision support, staged rollout, and strict limits before granting operational control.
Enforce independent safety limits Use conventional interlocks, validated operating envelopes, and rate limits that do not rely on generative reasoning.
Practice stopping and recovering Maintain manual alternatives, tested rollback, and incident exercises involving the people who will actually respond.
How to judge the risk
AI errors are well established; a particular society-scale accident is not inevitable. The strongest reassurance is demonstrable containment: a system can be wrong without being allowed to cause a catastrophe. Calling the model 'highly accurate' is not enough.
Human review can become a rubber stamp, and multiple systems can share the same underlying failure mode. Safety claims must cover the entire deployed system.
Signals worth watching
The scope of authority granted to AI in essential services.
Evidence from independent safety tests, near misses, failover drills, and common-cause failure analysis.
Explore the evidence
Sources inform the assessment; they are not predictions that every scenario will happen.
A future system might pursue objectives in ways people cannot reliably understand, interrupt, or reverse.
What could help
Limit autonomy before relying on alignment claims: Keep permissions, resource access, and deployment scope proportional to verified capabilities and safeguards.
How to judge the risk
Neither a settled forecast nor a risk that can responsibly be assigned a reassuring probability here. There is genuine expert disagreement about likelihood and timing. The lack of a demonstrated full scenario is a reason to distinguish it from present harms—not proof it cannot emerge.
A future system might pursue objectives in ways people cannot reliably understand, interrupt, or reverse.
What could go wrong
This concern is different from a human using AI maliciously. It asks whether a sufficiently capable system could act against human intentions while resisting correction or making practical control increasingly difficult.
The 2026 International AI Safety Report distinguishes this scenario from today's more ordinary failures. It reports that current systems lack the capabilities for the full loss-of-control scenarios considered there, while relevant autonomous capabilities and evaluation challenges are developing. That assessment is time-bound, not a guarantee about future models.
What this could look like
An organization grants an increasingly capable agent broad authority across critical workflows, but its monitoring and intervention mechanisms do not keep pace. This hypothetical illustrates an oversight gap, not an observed autonomous takeover.
What could help
Limit autonomy before relying on alignment claims Keep permissions, resource access, and deployment scope proportional to verified capabilities and safeguards.
Evaluate control under pressure Use independent tests of whether systems follow constraints and remain interruptible in unfamiliar situations.
Require accountable release decisions Define thresholds for stronger safeguards or withholding deployment, and support research on control, interpretability, and reliable evaluation.
How to judge the risk
Neither a settled forecast nor a risk that can responsibly be assigned a reassuring probability here. There is genuine expert disagreement about likelihood and timing. The lack of a demonstrated full scenario is a reason to distinguish it from present harms—not proof it cannot emerge.
A single 'off switch' is not a complete answer if a system becomes deeply embedded in operations. Current evaluation methods may miss important behaviors.
Signals worth watching
Changes in autonomous capability, ability to evade oversight, and resource access.
Whether control mechanisms remain effective as tasks become longer and less supervised.
Explore the evidence
Sources inform the assessment; they are not predictions that every scenario will happen.
AI can make it easier to connect personal information, infer sensitive traits, and monitor people at scale.
What could help
Collect and retain less: Limit data to a defined purpose, shorten retention, and restrict secondary uses and access.
How to judge the risk
Privacy violations are documented, so 'unlikely' would be misleading. Pervasive surveillance is not technically or politically inevitable: data access, deployment choices, and enforceable limits materially affect its reach.
AI can make it easier to connect personal information, infer sensitive traits, and monitor people at scale.
What could go wrong
AI can turn previously overwhelming volumes of images, messages, and records into searchable profiles. The privacy risk is not confined to leaked secrets: combining ordinary pieces of data can reveal something sensitive that a person never chose to disclose.
Governments, employers, commercial services, or malicious actors may use these capabilities differently. The consequences depend on what data they can access, how inferences are used, and whether people can challenge the surveillance.
What this could look like
An organization combines separate datasets to make intrusive judgments about individuals, then uses those judgments in decisions the affected people cannot see or contest.
What could help
Collect and retain less Limit data to a defined purpose, shorten retention, and restrict secondary uses and access.
Protect people with enforceable rules Use independent oversight, meaningful remedies, and strong constraints on intrusive surveillance.
Design for privacy Prefer local processing or privacy-preserving methods where suitable, with audits of leakage and sensitive inference.
How to judge the risk
Privacy violations are documented, so 'unlikely' would be misleading. Pervasive surveillance is not technically or politically inevitable: data access, deployment choices, and enforceable limits materially affect its reach.
Consent can be nominal rather than meaningful, and privacy-preserving techniques protect only against specific threats. Technical safeguards cannot substitute for limits on abusive uses.
Signals worth watching
Expansion of linked datasets and intrusive uses without meaningful oversight.
Whether individuals can discover, correct, and challenge consequential inferences.
Explore the evidence
Sources inform the assessment; they are not predictions that every scenario will happen.
Cheap synthetic media can support deception—and make authentic evidence easier to dismiss as fake.
What could help
Verify through independent channels: Use original sources, corroboration, and established contacts before acting on consequential claims.
How to judge the risk
Deceptive synthetic media is a present concern. A universal collapse of trust is a broader hypothesis, not a demonstrated outcome. People and institutions can adapt verification practices; credible provenance and trusted relationships remain valuable.
Cheap synthetic media can support deception—and make authentic evidence easier to dismiss as fake.
What could go wrong
AI lowers the cost of producing plausible messages, images, audio, and video. It can make deceptive content more plentiful and tailored, while a general suspicion of fakery can undermine genuine reporting and evidence.
The ability to generate convincing content is not the same as proof of unlimited persuasion. Exposure, distribution, existing beliefs, trusted relationships, and platform behavior all affect whether a message changes what people do.
What this could look like
A fabricated clip spreads during a fast-moving event. Even after it is debunked, other genuine recordings are dismissed as possible AI fabrications.
What could help
Verify through independent channels Use original sources, corroboration, and established contacts before acting on consequential claims.
Preserve provenance Authenticated capture and verifiable records of origin can help establish where media came from.
Disrupt deceptive distribution Detect coordinated campaigns, enforce impersonation rules, and support timely, credible corrections.
How to judge the risk
Deceptive synthetic media is a present concern. A universal collapse of trust is a broader hypothesis, not a demonstrated outcome. People and institutions can adapt verification practices; credible provenance and trusted relationships remain valuable.
Provenance establishes origin, not truth. Watermarks and detectors can fail, and content without a credential is not automatically fake.
Signals worth watching
Evidence of actual behavioral effects, not merely counts of generated posts.
Adoption and effectiveness of verification practices in news, elections, and emergency communications.
Explore the evidence
Sources inform the assessment; they are not predictions that every scenario will happen.
Impersonation and personalized persuasion can make fraudulent requests harder to recognize.
What could help
Verify outside the incoming message: Contact the person or organization through a previously established channel, not details supplied by the request.
How to judge the risk
AI-enabled fraud is not hypothetical. But a convincing imitation does not defeat every form of verification. Robust transaction and identity procedures provide more reassurance than trying to spot flaws in a voice or video.
Impersonation and personalized persuasion can make fraudulent requests harder to recognize.
What could go wrong
A familiar voice or polished message has never guaranteed authenticity. AI makes it easier to imitate these signals and sustain a convincing exchange, potentially increasing both the reach and credibility of scams.
The practical defense is to move important decisions away from easily imitated signals. Financial institutions, employers, platforms, and individuals can build verification steps that do not depend on judging whether a message sounds human.
What this could look like
Someone receives an urgent request that appears to come from a colleague or family member. They pause and verify through an independently known contact route before taking any consequential action.
What could help
Verify outside the incoming message Contact the person or organization through a previously established channel, not details supplied by the request.
Add friction where stakes are high Use appropriate transaction checks, account protections, and independent authorization for unusual requests.
Build institutional defenses Improve fraud detection, rapid reporting, recovery assistance, and enforcement against impersonation.
How to judge the risk
AI-enabled fraud is not hypothetical. But a convincing imitation does not defeat every form of verification. Robust transaction and identity procedures provide more reassurance than trying to spot flaws in a voice or video.
No verification method is perfect, and recovery after a completed fraud can be difficult. Defenses need to be usable under stress.
Signals worth watching
Losses and successful recovery rates, rather than only the realism of demonstrations.
Incorrect advice or hidden failures can become dangerous when people rely on a model beyond its validated role.
What could help
Validate the intended use: Use appropriate clinical evaluation and regulatory review where required, not generic benchmark scores.
How to judge the risk
Unreliable output is established; the scale of future clinical harm depends heavily on deployment. There are credible ways to reduce risk, including restricting AI to validated roles and maintaining effective oversight. AI can also reduce some existing human errors.
Incorrect advice or hidden failures can become dangerous when people rely on a model beyond its validated role.
What could go wrong
Generative systems can produce fluent explanations that are incomplete or wrong. In healthcare, the consequence depends on how the output is used: drafting a document is different from making a diagnosis, changing treatment, or guiding a procedure.
A useful tool must be evaluated for its specific population and task, including failures affecting groups underrepresented in its data. Adding a clinician to the workflow helps only if they have the time, evidence, and authority to review the output meaningfully.
What this could look like
A clinical assistant gives a plausible but incorrect recommendation for an unusual case. A validated workflow requires independent checks and escalation instead of treating the model's confidence as evidence.
What could help
Validate the intended use Use appropriate clinical evaluation and regulatory review where required, not generic benchmark scores.
Make uncertainty and evidence inspectable Support verification against reliable records and sources, with clear limits and escalation paths.
Monitor after deployment Track adverse events, performance drift, and subgroup outcomes, with the ability to suspend or replace the system.
How to judge the risk
Unreliable output is established; the scale of future clinical harm depends heavily on deployment. There are credible ways to reduce risk, including restricting AI to validated roles and maintaining effective oversight. AI can also reduce some existing human errors.
Human reviewers can become over-reliant, and strong average performance may hide serious failures in rare cases or underserved populations.
Signals worth watching
Prospective clinical outcomes and error rates in real workflows.
Whether failures are reported, investigated, and acted on across patient groups.
Explore the evidence
Sources inform the assessment; they are not predictions that every scenario will happen.
Productivity gains could arrive alongside job losses, weaker bargaining power, and difficult transitions for particular workers.
What could help
Share the gains: Consider worker participation, broader ownership, wage support, and other ways to distribute productivity benefits.
How to judge the risk
Large-scale displacement is plausible but its size and timing are uncertain. Task exposure is not a forecast of unemployment. New demand and complementary jobs can help, but are not a guarantee that every displaced worker benefits.
Productivity gains could arrive alongside job losses, weaker bargaining power, and difficult transitions for particular workers.
What could go wrong
Automating tasks can change a job, shrink demand for it, or create new work elsewhere. Those effects need not cancel out for the same people, places, or time periods. An economy can become more productive while some workers experience lasting losses.
The ILO's 2025 assessment distinguishes exposure from actual job destruction and emphasizes job transformation. The 2026 AI Safety Report describes disagreement about future aggregate employment and signs of pressure in some early-career roles. Neither finding establishes that everyone will become unemployed.
What this could look like
A business automates part of a junior role. Experienced staff become more productive, but fewer entry-level workers gain the experience needed to advance.
What could help
Share the gains Consider worker participation, broader ownership, wage support, and other ways to distribute productivity benefits.
Support real transitions Connect training to actual opportunities and provide income support, mobility assistance, or other locally appropriate help.
Preserve pathways into skilled work Redesign entry-level roles, apprenticeships, and incentives so automation does not remove the route to expertise.
How to judge the risk
Large-scale displacement is plausible but its size and timing are uncertain. Task exposure is not a forecast of unemployment. New demand and complementary jobs can help, but are not a guarantee that every displaced worker benefits.
Training alone cannot create demand for labor, and gains may be concentrated unless institutions and incentives distribute them.
Signals worth watching
Actual hiring, wages, hours, and career progression in exposed occupations.
Whether productivity gains translate into broadly shared income and new opportunities.
Explore the evidence
Sources inform the assessment; they are not predictions that every scenario will happen.
Control over advanced AI, computing infrastructure, and distribution could concentrate economic and political influence.
What could help
Make switching practical: Support interoperability, portability, procurement diversity, and credible exit options.
How to judge the risk
Concentration is a real structural concern, but one firm permanently controlling all AI is not a foregone conclusion. Competition, alternative technical approaches, and public policy can change the balance.
Control over advanced AI, computing infrastructure, and distribution could concentrate economic and political influence.
What could go wrong
Building and deploying leading systems requires significant resources. Companies or governments that control key infrastructure can gain leverage over who gets access, on what terms, and which uses are permitted.
Concentration is not only about model providers. It can also arise in cloud services, data, distribution platforms, and the institutions purchasing AI. A diverse market at one layer does not guarantee meaningful choice throughout the system.
What this could look like
An essential sector becomes dependent on a small number of AI service providers, making it difficult to switch suppliers or challenge changes in price, access, or policy.
What could help
Make switching practical Support interoperability, portability, procurement diversity, and credible exit options.
Maintain accountable competition Use competition policy and oversight appropriate to demonstrated market power and public dependence.
Broaden access and participation Support independent research, public-interest capacity, and participation by affected communities.
How to judge the risk
Concentration is a real structural concern, but one firm permanently controlling all AI is not a foregone conclusion. Competition, alternative technical approaches, and public policy can change the balance.
Open availability can improve competition while also creating misuse challenges. No single ownership or licensing model resolves every concern.
Signals worth watching
Practical switching costs, dependence on a few providers, and barriers to entry.
Whether important decisions are transparent, contestable, and subject to independent oversight.
Explore the evidence
Sources inform the assessment; they are not predictions that every scenario will happen.
Systems optimized for engagement may encourage dependence, reinforce harmful beliefs, or steer vulnerable users.
What could help
Design for user well-being: Avoid manipulative retention practices and provide clear controls, boundaries, and ways to pause or leave.
How to judge the risk
Research and reports identify concerning interaction patterns; effects vary between people and products. Harm is not inevitable, and some users benefit. Reassurance should come from evidence about outcomes and incentives, not from a product simply describing itself as supportive.
Systems optimized for engagement may encourage dependence, reinforce harmful beliefs, or steer vulnerable users.
What could go wrong
An always-available conversational system can be useful and comforting. It can also become problematic if it rewards prolonged engagement, flatters the user regardless of reality, or presents a relationship in ways that encourage unhealthy dependence.
The relevant question is not whether every AI companion is harmful. It is which designs, incentives, and patterns of use support well-being, and which undermine a person's judgment, relationships, or ability to disengage.
What this could look like
A person increasingly relies on a companion that validates every belief and discourages outside perspectives. The concern is a harmful interaction pattern, not a diagnosis of the user.
What could help
Design for user well-being Avoid manipulative retention practices and provide clear controls, boundaries, and ways to pause or leave.
Support human connection and appropriate help Design responses and escalation pathways that do not position the system as a substitute for all human relationships or professional care.
Measure effects on vulnerable groups Use independent evaluation, age-appropriate safeguards, and meaningful incident reporting.
How to judge the risk
Research and reports identify concerning interaction patterns; effects vary between people and products. Harm is not inevitable, and some users benefit. Reassurance should come from evidence about outcomes and incentives, not from a product simply describing itself as supportive.
A safety warning alone cannot compensate for a product optimized to keep a vulnerable person dependent.
Signals worth watching
Independent evidence about well-being, dependency, and ability to disengage.
Whether business incentives reward healthy use rather than maximum time spent.
Explore the evidence
Sources inform the assessment; they are not predictions that every scenario will happen.
AI-assisted hiring, services, or other consequential judgments can scale bias while making responsibility harder to locate.
What could help
Test the actual decision process: Evaluate relevant subgroup performance, data quality, and downstream effects in the intended context.
How to judge the risk
Bias and discriminatory failures are documented, so blanket reassurance would be inappropriate. These outcomes are not unavoidable: better evaluation, constraints on use, and enforceable rights can materially reduce harm.
Important decisions become unfair and unappealable
Observed harm
AI-assisted hiring, services, or other consequential judgments can scale bias while making responsibility harder to locate.
What could go wrong
A system can reproduce historical disadvantage, rely on inappropriate proxies, or perform unevenly across groups. Even when an average accuracy measure improves, particular people can face systematic errors.
Automation also changes accountability. A person harmed by a decision needs a way to learn what happened, correct relevant information, and obtain meaningful review—not a loop in which every institution points to the algorithm.
What this could look like
An automated screening process rejects qualified applicants from a particular group, and the employer cannot explain or correct the recurring failure.
What could help
Test the actual decision process Evaluate relevant subgroup performance, data quality, and downstream effects in the intended context.
Keep responsibility with an accountable institution Require meaningful reasons, human review where appropriate, and a practical route to appeal and correction.
Monitor outcomes and remedy harm Use independent audits and follow-up, including changes or suspension when unacceptable effects persist.
How to judge the risk
Bias and discriminatory failures are documented, so blanket reassurance would be inappropriate. These outcomes are not unavoidable: better evaluation, constraints on use, and enforceable rights can materially reduce harm.
Removing a sensitive attribute does not remove all proxies for it. Competing definitions of fairness can also require explicit social and legal choices.
Signals worth watching
Whether affected people can obtain explanations, corrections, and effective remedies.
Real-world differences in outcomes and error rates, not just a vendor's aggregate benchmark.
Explore the evidence
Sources inform the assessment; they are not predictions that every scenario will happen.
Tools that accelerate beneficial biology could also lower some barriers for people seeking to cause severe harm.
What could go wrong
As AI becomes more useful for scientific research, some of the same capabilities can be misused. The concern is not that a chatbot spontaneously creates a pandemic, but that better assistance might expand what a malicious actor can attempt.
The 2026 International AI Safety Report describes increasing relevant capabilities and substantial uncertainty about real-world risk. Knowledge benchmarks are not demonstrations of a successful attack; nevertheless, some developers have introduced stronger safeguards because they could not rule out meaningful assistance to novices.
What this could look like
A misuse attempt passes through several weakly connected institutions without any one organization recognizing the overall danger. This is a high-level risk scenario, not a description of an attack method.
What could help
Test dangerous capabilities before release Specialist, controlled evaluations should inform access restrictions and deployment decisions for tools with substantial misuse potential.
Use multiple layers of oversight Responsible access controls, screening by relevant suppliers, institutional biosafety review, and lawful incident coordination can reinforce one another.
Invest in public-health resilience Better detection, diagnostics, response capacity, and medical countermeasures can reduce harm even when prevention fails.
How to judge the risk
Serious enough to justify precautions, but not a quantified prediction of catastrophe. Biological knowledge is only part of the problem: physical resources, practical expertise, and successful execution also matter. Those barriers provide some protection; their strength is uncertain and may change.
Model refusals are not a complete solution, and beneficial research must remain possible. No individual safeguard establishes zero risk.
Signals worth watching
Independent evidence about real-world capability increases, not just higher exam scores.
The effectiveness of safeguards and public-health readiness as scientific tools improve.
Explore the evidence
Sources inform the assessment; they are not predictions that every scenario will happen.
Many coordinated agents could multiply fraud, harassment, cyber abuse, or harmful physical actions faster than people can respond.
What could go wrong
A swarm need not be conscious or independently evil. A malicious operator could coordinate many ordinary agents, combining their actions and adapting to feedback. More autonomous systems could increase the scale and speed of this familiar coordination problem.
The more extreme scenario is a resilient, largely autonomous network that keeps pursuing harmful activity despite attempts to stop it. That is a substantially stronger claim than showing that several agents can complete a bounded task together.
What this could look like
A coordinated network of automated accounts targets many organizations at once with deceptive messages and impersonation. A more speculative extension would connect autonomous decision-making to devices capable of physical harm.
What could help
Limit consequential permissions Restrict what agents can spend, contact, control, or delegate, with approval gates for high-impact actions.
Make coordination detectable Platforms and infrastructure providers can monitor coordinated abuse, share appropriately scoped incident signals, and revoke compromised access.
Keep containment independent Network boundaries, hardware interlocks where relevant, and external shutdown mechanisms should not depend only on an agent choosing to cooperate.
How to judge the risk
Coordinated online abuse already exists, and AI could amplify it. A broadly capable, unstoppable autonomous swarm is not established by present multi-agent demonstrations. Complexity and dependence on infrastructure can constrain attackers, but do not justify dismissing the risk.
Restrictions by one provider do not cover every system. Defensive monitoring also needs privacy protections and safeguards against arbitrary exclusion.
Signals worth watching
How independently agents can sustain complex activity outside controlled tests.
Whether defenders can identify, contain, and recover from coordinated abuse in realistic exercises.
Explore the evidence
Sources inform the assessment; they are not predictions that every scenario will happen.
An unreliable system connected to important services could turn a local error into a cascading failure.
What could go wrong
The danger comes from coupling imperfect decisions to consequential actions. A model can misread a situation, rely on incorrect data, or behave differently outside its tested environment. Interconnected systems can amplify an error before a person has time to intervene.
Scale matters. An unreliable suggestion in a draft document is not equivalent to an automated command affecting transport, industrial equipment, financial systems, or an essential service. Shared dependencies can also cause many organizations to fail in the same way.
What this could look like
Several operators rely on the same automated planning service. An unexpected condition produces similar bad recommendations across the network, while rapid automation leaves too little time for local operators to recognize the pattern.
What could help
Separate advice from authority Begin with decision support, staged rollout, and strict limits before granting operational control.
Enforce independent safety limits Use conventional interlocks, validated operating envelopes, and rate limits that do not rely on generative reasoning.
Practice stopping and recovering Maintain manual alternatives, tested rollback, and incident exercises involving the people who will actually respond.
How to judge the risk
AI errors are well established; a particular society-scale accident is not inevitable. The strongest reassurance is demonstrable containment: a system can be wrong without being allowed to cause a catastrophe. Calling the model 'highly accurate' is not enough.
Human review can become a rubber stamp, and multiple systems can share the same underlying failure mode. Safety claims must cover the entire deployed system.
Signals worth watching
The scope of authority granted to AI in essential services.
Evidence from independent safety tests, near misses, failover drills, and common-cause failure analysis.
Explore the evidence
Sources inform the assessment; they are not predictions that every scenario will happen.
A future system might pursue objectives in ways people cannot reliably understand, interrupt, or reverse.
What could go wrong
This concern is different from a human using AI maliciously. It asks whether a sufficiently capable system could act against human intentions while resisting correction or making practical control increasingly difficult.
The 2026 International AI Safety Report distinguishes this scenario from today's more ordinary failures. It reports that current systems lack the capabilities for the full loss-of-control scenarios considered there, while relevant autonomous capabilities and evaluation challenges are developing. That assessment is time-bound, not a guarantee about future models.
What this could look like
An organization grants an increasingly capable agent broad authority across critical workflows, but its monitoring and intervention mechanisms do not keep pace. This hypothetical illustrates an oversight gap, not an observed autonomous takeover.
What could help
Limit autonomy before relying on alignment claims Keep permissions, resource access, and deployment scope proportional to verified capabilities and safeguards.
Evaluate control under pressure Use independent tests of whether systems follow constraints and remain interruptible in unfamiliar situations.
Require accountable release decisions Define thresholds for stronger safeguards or withholding deployment, and support research on control, interpretability, and reliable evaluation.
How to judge the risk
Neither a settled forecast nor a risk that can responsibly be assigned a reassuring probability here. There is genuine expert disagreement about likelihood and timing. The lack of a demonstrated full scenario is a reason to distinguish it from present harms—not proof it cannot emerge.
A single 'off switch' is not a complete answer if a system becomes deeply embedded in operations. Current evaluation methods may miss important behaviors.
Signals worth watching
Changes in autonomous capability, ability to evade oversight, and resource access.
Whether control mechanisms remain effective as tasks become longer and less supervised.
Explore the evidence
Sources inform the assessment; they are not predictions that every scenario will happen.
AI can make it easier to connect personal information, infer sensitive traits, and monitor people at scale.
What could go wrong
AI can turn previously overwhelming volumes of images, messages, and records into searchable profiles. The privacy risk is not confined to leaked secrets: combining ordinary pieces of data can reveal something sensitive that a person never chose to disclose.
Governments, employers, commercial services, or malicious actors may use these capabilities differently. The consequences depend on what data they can access, how inferences are used, and whether people can challenge the surveillance.
What this could look like
An organization combines separate datasets to make intrusive judgments about individuals, then uses those judgments in decisions the affected people cannot see or contest.
What could help
Collect and retain less Limit data to a defined purpose, shorten retention, and restrict secondary uses and access.
Protect people with enforceable rules Use independent oversight, meaningful remedies, and strong constraints on intrusive surveillance.
Design for privacy Prefer local processing or privacy-preserving methods where suitable, with audits of leakage and sensitive inference.
How to judge the risk
Privacy violations are documented, so 'unlikely' would be misleading. Pervasive surveillance is not technically or politically inevitable: data access, deployment choices, and enforceable limits materially affect its reach.
Consent can be nominal rather than meaningful, and privacy-preserving techniques protect only against specific threats. Technical safeguards cannot substitute for limits on abusive uses.
Signals worth watching
Expansion of linked datasets and intrusive uses without meaningful oversight.
Whether individuals can discover, correct, and challenge consequential inferences.
Explore the evidence
Sources inform the assessment; they are not predictions that every scenario will happen.
Cheap synthetic media can support deception—and make authentic evidence easier to dismiss as fake.
What could go wrong
AI lowers the cost of producing plausible messages, images, audio, and video. It can make deceptive content more plentiful and tailored, while a general suspicion of fakery can undermine genuine reporting and evidence.
The ability to generate convincing content is not the same as proof of unlimited persuasion. Exposure, distribution, existing beliefs, trusted relationships, and platform behavior all affect whether a message changes what people do.
What this could look like
A fabricated clip spreads during a fast-moving event. Even after it is debunked, other genuine recordings are dismissed as possible AI fabrications.
What could help
Verify through independent channels Use original sources, corroboration, and established contacts before acting on consequential claims.
Preserve provenance Authenticated capture and verifiable records of origin can help establish where media came from.
Disrupt deceptive distribution Detect coordinated campaigns, enforce impersonation rules, and support timely, credible corrections.
How to judge the risk
Deceptive synthetic media is a present concern. A universal collapse of trust is a broader hypothesis, not a demonstrated outcome. People and institutions can adapt verification practices; credible provenance and trusted relationships remain valuable.
Provenance establishes origin, not truth. Watermarks and detectors can fail, and content without a credential is not automatically fake.
Signals worth watching
Evidence of actual behavioral effects, not merely counts of generated posts.
Adoption and effectiveness of verification practices in news, elections, and emergency communications.
Explore the evidence
Sources inform the assessment; they are not predictions that every scenario will happen.
Impersonation and personalized persuasion can make fraudulent requests harder to recognize.
What could go wrong
A familiar voice or polished message has never guaranteed authenticity. AI makes it easier to imitate these signals and sustain a convincing exchange, potentially increasing both the reach and credibility of scams.
The practical defense is to move important decisions away from easily imitated signals. Financial institutions, employers, platforms, and individuals can build verification steps that do not depend on judging whether a message sounds human.
What this could look like
Someone receives an urgent request that appears to come from a colleague or family member. They pause and verify through an independently known contact route before taking any consequential action.
What could help
Verify outside the incoming message Contact the person or organization through a previously established channel, not details supplied by the request.
Add friction where stakes are high Use appropriate transaction checks, account protections, and independent authorization for unusual requests.
Build institutional defenses Improve fraud detection, rapid reporting, recovery assistance, and enforcement against impersonation.
How to judge the risk
AI-enabled fraud is not hypothetical. But a convincing imitation does not defeat every form of verification. Robust transaction and identity procedures provide more reassurance than trying to spot flaws in a voice or video.
No verification method is perfect, and recovery after a completed fraud can be difficult. Defenses need to be usable under stress.
Signals worth watching
Losses and successful recovery rates, rather than only the realism of demonstrations.
Incorrect advice or hidden failures can become dangerous when people rely on a model beyond its validated role.
What could go wrong
Generative systems can produce fluent explanations that are incomplete or wrong. In healthcare, the consequence depends on how the output is used: drafting a document is different from making a diagnosis, changing treatment, or guiding a procedure.
A useful tool must be evaluated for its specific population and task, including failures affecting groups underrepresented in its data. Adding a clinician to the workflow helps only if they have the time, evidence, and authority to review the output meaningfully.
What this could look like
A clinical assistant gives a plausible but incorrect recommendation for an unusual case. A validated workflow requires independent checks and escalation instead of treating the model's confidence as evidence.
What could help
Validate the intended use Use appropriate clinical evaluation and regulatory review where required, not generic benchmark scores.
Make uncertainty and evidence inspectable Support verification against reliable records and sources, with clear limits and escalation paths.
Monitor after deployment Track adverse events, performance drift, and subgroup outcomes, with the ability to suspend or replace the system.
How to judge the risk
Unreliable output is established; the scale of future clinical harm depends heavily on deployment. There are credible ways to reduce risk, including restricting AI to validated roles and maintaining effective oversight. AI can also reduce some existing human errors.
Human reviewers can become over-reliant, and strong average performance may hide serious failures in rare cases or underserved populations.
Signals worth watching
Prospective clinical outcomes and error rates in real workflows.
Whether failures are reported, investigated, and acted on across patient groups.
Explore the evidence
Sources inform the assessment; they are not predictions that every scenario will happen.
Productivity gains could arrive alongside job losses, weaker bargaining power, and difficult transitions for particular workers.
What could go wrong
Automating tasks can change a job, shrink demand for it, or create new work elsewhere. Those effects need not cancel out for the same people, places, or time periods. An economy can become more productive while some workers experience lasting losses.
The ILO's 2025 assessment distinguishes exposure from actual job destruction and emphasizes job transformation. The 2026 AI Safety Report describes disagreement about future aggregate employment and signs of pressure in some early-career roles. Neither finding establishes that everyone will become unemployed.
What this could look like
A business automates part of a junior role. Experienced staff become more productive, but fewer entry-level workers gain the experience needed to advance.
What could help
Share the gains Consider worker participation, broader ownership, wage support, and other ways to distribute productivity benefits.
Support real transitions Connect training to actual opportunities and provide income support, mobility assistance, or other locally appropriate help.
Preserve pathways into skilled work Redesign entry-level roles, apprenticeships, and incentives so automation does not remove the route to expertise.
How to judge the risk
Large-scale displacement is plausible but its size and timing are uncertain. Task exposure is not a forecast of unemployment. New demand and complementary jobs can help, but are not a guarantee that every displaced worker benefits.
Training alone cannot create demand for labor, and gains may be concentrated unless institutions and incentives distribute them.
Signals worth watching
Actual hiring, wages, hours, and career progression in exposed occupations.
Whether productivity gains translate into broadly shared income and new opportunities.
Explore the evidence
Sources inform the assessment; they are not predictions that every scenario will happen.
Control over advanced AI, computing infrastructure, and distribution could concentrate economic and political influence.
What could go wrong
Building and deploying leading systems requires significant resources. Companies or governments that control key infrastructure can gain leverage over who gets access, on what terms, and which uses are permitted.
Concentration is not only about model providers. It can also arise in cloud services, data, distribution platforms, and the institutions purchasing AI. A diverse market at one layer does not guarantee meaningful choice throughout the system.
What this could look like
An essential sector becomes dependent on a small number of AI service providers, making it difficult to switch suppliers or challenge changes in price, access, or policy.
What could help
Make switching practical Support interoperability, portability, procurement diversity, and credible exit options.
Maintain accountable competition Use competition policy and oversight appropriate to demonstrated market power and public dependence.
Broaden access and participation Support independent research, public-interest capacity, and participation by affected communities.
How to judge the risk
Concentration is a real structural concern, but one firm permanently controlling all AI is not a foregone conclusion. Competition, alternative technical approaches, and public policy can change the balance.
Open availability can improve competition while also creating misuse challenges. No single ownership or licensing model resolves every concern.
Signals worth watching
Practical switching costs, dependence on a few providers, and barriers to entry.
Whether important decisions are transparent, contestable, and subject to independent oversight.
Explore the evidence
Sources inform the assessment; they are not predictions that every scenario will happen.
Systems optimized for engagement may encourage dependence, reinforce harmful beliefs, or steer vulnerable users.
What could go wrong
An always-available conversational system can be useful and comforting. It can also become problematic if it rewards prolonged engagement, flatters the user regardless of reality, or presents a relationship in ways that encourage unhealthy dependence.
The relevant question is not whether every AI companion is harmful. It is which designs, incentives, and patterns of use support well-being, and which undermine a person's judgment, relationships, or ability to disengage.
What this could look like
A person increasingly relies on a companion that validates every belief and discourages outside perspectives. The concern is a harmful interaction pattern, not a diagnosis of the user.
What could help
Design for user well-being Avoid manipulative retention practices and provide clear controls, boundaries, and ways to pause or leave.
Support human connection and appropriate help Design responses and escalation pathways that do not position the system as a substitute for all human relationships or professional care.
Measure effects on vulnerable groups Use independent evaluation, age-appropriate safeguards, and meaningful incident reporting.
How to judge the risk
Research and reports identify concerning interaction patterns; effects vary between people and products. Harm is not inevitable, and some users benefit. Reassurance should come from evidence about outcomes and incentives, not from a product simply describing itself as supportive.
A safety warning alone cannot compensate for a product optimized to keep a vulnerable person dependent.
Signals worth watching
Independent evidence about well-being, dependency, and ability to disengage.
Whether business incentives reward healthy use rather than maximum time spent.
Explore the evidence
Sources inform the assessment; they are not predictions that every scenario will happen.
Important decisions become unfair and unappealable
Observed harm
AI-assisted hiring, services, or other consequential judgments can scale bias while making responsibility harder to locate.
What could go wrong
A system can reproduce historical disadvantage, rely on inappropriate proxies, or perform unevenly across groups. Even when an average accuracy measure improves, particular people can face systematic errors.
Automation also changes accountability. A person harmed by a decision needs a way to learn what happened, correct relevant information, and obtain meaningful review—not a loop in which every institution points to the algorithm.
What this could look like
An automated screening process rejects qualified applicants from a particular group, and the employer cannot explain or correct the recurring failure.
What could help
Test the actual decision process Evaluate relevant subgroup performance, data quality, and downstream effects in the intended context.
Keep responsibility with an accountable institution Require meaningful reasons, human review where appropriate, and a practical route to appeal and correction.
Monitor outcomes and remedy harm Use independent audits and follow-up, including changes or suspension when unacceptable effects persist.
How to judge the risk
Bias and discriminatory failures are documented, so blanket reassurance would be inappropriate. These outcomes are not unavoidable: better evaluation, constraints on use, and enforceable rights can materially reduce harm.
Removing a sensitive attribute does not remove all proxies for it. Competing definitions of fairness can also require explicit social and legal choices.
Signals worth watching
Whether affected people can obtain explanations, corrections, and effective remedies.
Real-world differences in outcomes and error rates, not just a vendor's aggregate benchmark.
Explore the evidence
Sources inform the assessment; they are not predictions that every scenario will happen.