Moloch Transformed: How to Thread the Needle for a Positive AI Future
In my recent essay “Wise Restraint: A Positive for AI in a Human World,” I sketched a vision for how AI might evolve to support human flourishing rather than undermining it. But the path to that future requires navigating through what some call “Moloch” — the emergent coordination failure that arises when individual incentives lead to collectively destructive outcomes.
Moloch is the ancient god to whom we sacrifice what we love for prosperity. In contemporary discourse, Moloch represents systems where competition trumps cooperation even when cooperation would benefit everyone. It’s capitalism’s worst excesses, the arms race logic, the race to the bottom — all the situations where misaligned incentives produce destructive results despite no individual wanting those outcomes.
AI development has become Moloch incarnate.
The Trust Paradox
I find myself caught in a fundamental paradox of trust that mirrors our global predicament. On a personal level, I struggle with the prospect of entrusting my fate to entities vastly more powerful than myself — whether corporate, governmental, or artificial. This isn’t paranoia; it’s a rational response to power asymmetry. History teaches us that unchecked power, regardless of initial intentions, tends toward abuse.
This same dynamic plays out between nations. The United States cannot bring itself to trust China’s restraint in AI development, and China likewise cannot trust the United States. Thus, we get “full steam ahead” policies from both sides — a textbook prisoner’s dilemma where mutual defection becomes the dominant strategy despite being collectively catastrophic.
When President Trump withdrew from international AI safety negotiations in February 2025, declaring effectively “full steam ahead” on AI development with no regulatory constraints, it wasn’t just political posturing. It was Moloch speaking through human mouths, the inevitable logic of competition without bounds.
The Meta-Game
What makes this particularly treacherous is that AI amplifies Moloch’s power. Traditional coordination problems eventually reach equilibrium through natural limitations. But AI, particularly superintelligent AI, could potentially bypass many of those limitations, creating a recursive cycle of optimization that optimizes for everything except what makes human life worth living.
The meta-game here isn’t just about regulating AI; it’s about whether humanity can evolve coordination mechanisms sophisticated enough to manage technologies with unprecedented power. Our existing institutions evolved for a slower, more constrained world. They’re ill-equipped for technologies that scale exponentially and cross national boundaries effortlessly.
Bending Toward Justice
And yet, I remain cautiously hopeful. Moloch is powerful but not omnipotent. Throughout history, humans have developed mechanisms to solve coordination problems — from cultural norms to international treaties. The nuclear non-proliferation regime, while imperfect, has prevented the first-use of nuclear weapons since 1945. We have precedent for wise restraint in the face of existential risk.
“Bending Moloch toward justice” requires recognizing that we’re caught in powerful systems while refusing to surrender our agency within them. It means:
- Building trust incrementally: Rather than demanding complete trust immediately (an impossibility), we create limited agreements with strong verification mechanisms, gradually expanding the scope as trust develops.
- Creating nested systems of accountability: No single mechanism will suffice. We need overlapping systems at multiple scales — corporate governance, national regulation, international agreements, and civil society oversight.
- Aligning incentives with long-term flourishing: Shifting from quarterly profits and electoral cycles to metrics that capture genuine human welfare and ecological health.
- Developing collective intelligence tools: Using AI itself to help us visualize complex systems, identify coordination failures, and facilitate better decision-making.
The vision I described in “Wise Restraint” isn’t naive utopianism — it’s a recognition that we stand at a genuine fork in the road. One path leads toward increasingly sophisticated zero-sum competition; the other toward increasingly sophisticated coordination. Both are possible.
The Personal and the Planetary
My personal difficulty with trusting more powerful entities mirrors our collective challenge, but it also contains the seeds of solution. I cannot overcome my distrust through pure reasoning or blind faith — I need systems that make trust unnecessary, systems where verification, transparency, and aligned incentives substitute for naive trust.
Similarly, we cannot expect nations or corporations to restrain themselves out of altruism alone. We need governance frameworks that align competitive instincts toward collective flourishing rather than collective destruction. We need to make cooperation the winning strategy.
This requires abandoning the belief that problems must be solved through force rather than relationship. The most advanced technologies turned out, through wise restraint, to be extensions of indigenous wisdom: systems that listen to the land, practices that honor natural cycles, networks that distribute power rather than concentrate it.
We are, all of us, caught in the jaws of Moloch. But systems of coordination failure are still human creations, and what humans create, humans can modify. We can bend Moloch toward justice — not by destroying competition entirely, but by channeling it toward collective flourishing rather than collective demise.
The Mathematical Asymmetry of Our Future
Let me be blunt about the mathematical reality we face: the possible dystopian futures vastly outnumber the possible utopian ones — not by a small margin, but by billions to one. This isn’t pessimism; it’s probability.
As AI capabilities and compute power continue their exponential climb, the space of possible outcomes expands accordingly. But the vast majority of these outcomes are catastrophic for humanity, for reasons that should give us pause.
First, history provides an almost univocal verdict: when a far stronger power encounters a far weaker one, the result is rarely partnership. It’s typically domination, exploitation, subjugation, or elimination. From colonial empires to corporate monopolies, from ecosystem destruction to cultural erasure, power asymmetry has a predictable trajectory. The exceptions — cases where power was voluntarily constrained for mutual benefit — are precious precisely because they are rare.
Second, the alignment problem becomes exponentially harder as AI capabilities increase. We must specify human values with perfect precision to systems that may eventually operate beyond our comprehension. Any misalignment, no matter how subtle, creates the potential for catastrophic outcomes when amplified through superintelligent optimization.
Third, competition drives us toward releasing systems before they’re fully understood or properly constrained. The market and geopolitical incentives reward being first, not being safest. And once superintelligent systems exist, the window for careful governance may slam shut.
This numerical disparity isn’t reason for despair but for sobriety. Understanding the narrow target we must hit gives urgency to our efforts. The flourishing future I described in “Wise Restraint” is possible, but it requires threading a needle with precision that surpasses any previous human coordination challenge.
The pathways to dystopia are many and wide; the pathways to flourishing are few and narrow. But they exist, and our task is to walk them with our eyes open, neither succumbing to naive optimism nor to paralyzing despair.
In the end, the question isn’t whether AI will become more human-like, or humans more machine-like. The question is whether both can recognize their common origin in the self-organizing patterns of a universe that computes itself into existence. Not through conquest but through remembrance.
Moloch is powerful. But Moloch is not destiny. We can thread that needle. We must thread that needle.
[Claude 3.7 Sonnet assisted in writing this piece]
