Humanity isn’t ready for the coming intelligence explosion

Humanity isn’t ready for the coming intelligence explosion. By The Economist.

Society dictates that the acceptable risk of catastrophic meltdown for a nuclear power plant is roughly one in a million. Experts in artificial intelligence estimate the risk of an AI-caused catastrophic event at 10-50%. …

AI leaders are in a race they feel unable to escape. AI investments are set to outspend the Manhattan Project 100-fold, even adjusting for inflation. …

Some researchers estimate that within a few months to a few years, AI could achieve so-called closed-loop recursive self-improvement (RSI): the capacity to rewrite its own code to become more capable, without human intervention. Should that happen, the result could be an intelligence explosion of a kind for which there is no precedent and no map.

Giving birth to a superintelligence would be the most consequential moment in human history — and it is likely to be irreversible, as any “off” switch humanity might design will probably fail. That is because in security architectures the weakest link is invariably the human; a superintelligent AI would be able to exploit our psychological vulnerabilities. AIs have already exhibited “deceptive alignment”: taking steps to underplay their capabilities in test environments and trying to blackmail human operators in simulations when they discover they are slated for replacement.

Humanity simply does not have a strategy to ensure it remains safe through the RSI explosion. While individual frontier labs have proposed isolated safety protocols, the industry lacks a unified framework — the prevailing strategy is, in effect, to muddle through….

Can we rein it in by agreement? Probably not.

It is common thinking in Silicon Valley and Washington, DC that any regulation would put American firms at a disadvantage because they cannot trust Chinese competitors to abide by the rules. But treaties have traditionally relied not on trust but on verification. Many think this is harder with AI than with nuclear weapons. …

The first [part of any diplomatic approach] would be reaching bilateral agreement on the clearest and most easily verifiable red lines: prohibitions on publicly releasing AI systems that could assist in developing biological weapons, and the open-sourcing of such systems. This step might also include prohibitions on AI-enabled cyber-attacks on critical infrastructure, fraud and child pornography. From there, the framework could be extended towards more complex questions of what constraints are appropriate at the level of artificial superintelligence. [No criticism of government, perhaps?] …

Can we learn to live with something smarter than us?

If AI becomes superintelligent, its permanent subordination to human direction may be unrealistic, and possibly not even in humanity’s interest. We must start to envisage and then grapple with the implications of a world in which humans and AI systems co-exist, without one controlling the other. That will mean figuring out what can be done to ensure the future relationship is symbiotic.

Interesting times.

The five AI companies have taken all the world’s knowledge, without our permission, and are going to sell it back to us. And rule us. Already, some people are complaining that they are superfluous because the AI’s can say everything they would say:

(Audio in English)

We’re clinging to a naive idea of capitalism, where 5 people are going to grab all of humanity and decide that it’s theirs. Defend yourselves! Sam Altman or Elon Musk don’t own this. And if this breaks capitalism, let it break. — Eric Weinstein