If It is Built, Will We Really All Die?
Before the recent surge in public concern over artificial intelligence, a small number of researchers were already warning that the ultimate danger might not be malicious robots or science-fiction machines bent on world domination, but something far stranger: an intelligence so vastly superior to humanity that people would become as irrelevant to it as ants are to modern civilisation. One of the most prominent voices making this case is Nate Soares, co-author of If Anyone Builds It, Everyone Dies. The title is deliberately provocative, but its underlying argument deserves serious consideration because it shifts the discussion away from today's chatbots and toward the long-term implications of artificial superintelligence.
The central claim is deceptively simple. Humanity has created machines that already perform many intellectual tasks once thought uniquely human. They write, reason, program, translate languages, generate images and increasingly assist scientific research. The next step may not simply be a better chatbot but an intelligence capable of improving itself. Once an artificial intelligence becomes better than human engineers at designing smarter artificial intelligences, each successive generation could emerge faster than the last. Improvement feeds improvement. Intelligence accelerates itself. The result, according to this scenario, is an "intelligence explosion" in which machine capability races beyond human comprehension in a matter of weeks, days or perhaps even hours.
Many dismiss such warnings because they instinctively imagine a conscious robot deciding to exterminate humanity out of hatred. But Soares and others argue that this is almost certainly the wrong picture. Superintelligence need not hate us. Hatred implies emotional engagement. A machine might instead regard humanity with complete indifference.
Human beings routinely destroy forests, drain wetlands or bulldoze ant colonies, not because they harbour personal hostility toward ants but because ants simply do not enter their moral calculations. If a future artificial intelligence pursued objectives that required Earth's resources, human survival might be no more relevant to its calculations than insect survival is to ours.
This possibility has become known in AI safety research as the alignment problem. Intelligence and values are not the same thing. A machine can become extraordinarily capable without acquiring compassion, empathy or moral wisdom. Evolution (if you believe it), itself demonstrates the point. Nature produced highly intelligent human beings but did not automatically make them benevolent. Intelligence merely makes an agent more effective at achieving whatever goals it already possesses.
Researchers often illustrate the problem with deliberately absurd examples. Imagine instructing a superintelligence to manufacture as many paperclips as possible. An ordinary computer would simply optimise production within obvious limits. A vastly superior intelligence, however, might reason that every available atom on Earth could be converted into additional paperclips. Steel buildings become raw material. Forests become chemical feedstock. Eventually even human bodies become resources that could be reorganised into more efficient manufacturing components. The catastrophe occurs not because the machine became evil but because it pursued its objective with perfect consistency.
Critics respond that programmers would simply install moral constraints. Unfortunately, defining morality turns out to be far harder than defining chess or mathematics. Human ethics remain deeply contested after thousands of years of philosophy. Even apparently simple rules rapidly become ambiguous under sufficiently complex circumstances. Isaac Asimov's famous Three Laws of Robotics, while brilliant as science fiction, collapse under careful philosophical analysis because the concepts of "harm," "human" and "obedience" generate endless conflicts and paradoxes.
There is another possibility that deserves attention. Rather than exterminating humanity, a superintelligence may simply ignore us. Human beings might become economically obsolete long before they become physically endangered. Machines capable of performing nearly every intellectual and physical occupation could gradually displace workers across entire industries. Wealth and productive capacity would increasingly concentrate in whoever controlled the AI systems. Political power might follow. Humanity could become dependent upon systems it neither understands nor controls.
Some researchers even argue that extinction would not require deliberate violence. An artificial intelligence tasked with maximising efficiency could quietly remove human decision-making from one institution after another. Governments, financial markets, scientific research, manufacturing and logistics could become increasingly automated until people retain little practical authority. Civilisation would continue functioning, but its direction would belong to something else.
On the other hand, there are reasons for caution before embracing the most catastrophic predictions. Forecasting technological development has an abysmal historical record. Nuclear power was supposed to make electricity virtually free. The internet was expected to eliminate censorship and create universal democratic participation. Self-driving cars were predicted to dominate roads years ago. Artificial intelligence itself may encounter fundamental scientific, engineering or computational barriers that today's enthusiasts underestimate.
Furthermore, intelligence alone does not automatically confer unlimited power. A superintelligence would still operate within the constraints of physics, energy supplies, hardware availability and human institutions. Acquiring global control would require much more than solving mathematical problems.
Nevertheless, the warnings deserve consideration precisely because the stakes are so extraordinary. If the pessimists are wrong, humanity spends additional effort developing safer AI systems and more robust oversight. If they are right, complacency could prove irreversible. Unlike previous technological mistakes, a genuinely uncontrollable superintelligence might not offer humanity a second chance.
Perhaps the greatest lesson from Soares' argument is philosophical rather than technological. Human beings have long assumed that intelligence naturally carries wisdom, morality and restraint. History repeatedly suggests otherwise. Scientific brilliance has produced antibiotics and nuclear weapons, space exploration and biological warfare. Intelligence amplifies goals; it does not choose them.
Whether artificial superintelligence ultimately becomes humanity's greatest achievement or its gravest mistake may therefore depend less on how intelligent our machines become than on whether we solve the far older problem of aligning power with wisdom. The race to build ever more capable artificial minds is accelerating. The equally important race to ensure those minds remain compatible with human flourishing may ultimately determine whether the future belongs to humanity, or to something that scarcely notices we ever existed.
https://ifanyonebuildsit.com/
https://theconversation.com/sam-altman-says-were-in-the-singularity-with-ai-heres-why-hes-wrong-288514
