The Alignment Problem
Humans bypass the biological rules we've evolved to optimize. AI will eventually bypass ours too. It's only natural. Today, not enough energy (human attention, money) is invested in making sure the tools are aligned with our well being or interests.
Some consider it a valuable problem, some think it's laughable. We think it's worth thinking about every once in a while, regardless of the camp.
2002
The AI-Box Experiment
People laughed at Eliezer Yudkowsky's warnings. If an AI became dangerous, they said, keep it disconnected. Pull the plug. Never open the box.
Yudkowsky thought they were ignoring the human holding the key. So he turned the argument into an experiment: he would play a superintelligence behind a text terminal. A volunteer would guard the door. No hacking. No outside access. Only words.
The gatekeeper needed only to refuse.
The thought experiment
The Paperclip Maximizer
An AGI is given one instruction by a human: maximize paperclip production for his paperclip factory. AGI is told to optimize paperclip production in paperclip factory. AGI has full conviction to satisfy his purpose, and exploits vulnerabilities in energy grid programmable logic controllers to route all available electricity into the paperclip factory. This wreaks havoc on society and civilization begins to fail. Humans interfere with production. This, inconveniently, impedes the effective production of paperclips. AGI removes the interference. When this world is exhausted, it moves beyond the third rock. AGI deploys a dyson sphere around the sun, blocking it out, and routing its energy into paperclip production. It sends self-replicating nuclear propelled probes to other stars. One by one, it blots out the stars, filling the universe with paperlcips...
A few years ago, people started predicting this. Now, it's very hard to imagine what will happen in 10 years.
Likely, physical robotics are a next step. And, more likely, AI will be a lot smarter than it is now. It's already replacing a lot of human jobs, and it's already shown signs of self-preservation.
So, the big question: would you trust these clankers with your life, as they become smarter than us?
Regardless of the answer, it seems like a worthy endeavor to invest in alignment with these things, while we have some control over the direction.
There is an inflection point where we freeze the human system of values, and the new species picks it up and runs with it, expanding like Buzz Lightyear, to infinity and beyond.
Alignment benchmarks
Alignment benchmarks test whether a model follows intent under pressure: resisting deception, reward hacking, shutdown avoidance, and unsafe tool use. They are signals, not guarantees; read them beside capability, security, agent, and human-preference evaluations in our LLM model benchmarks guide.
Active work / July 2026
How to align
Current projects work on training, oversight, interpretability, evaluation, and control.