"GLOBAL ISSUES, LEADERSHIP CHOICES":

Most of the concern about "big tech" seems to focus on economic power and privacy. This article written by two Oxford University scholars working at the nexus of philosophy and international affairs raises a different question: what should be be doing to prevent the technological development that might lead to our destruction?

"One way of looking at human creativity is as a process of pulling balls out of a giant urn. The balls represent ideas, discoveries and inventions. Over the course of history, we have extracted many balls. Most have been beneficial to humanity. The rest have been various shades of grey: a mix of good and bad, whose net effect is difficult to estimate. What we haven’t pulled out yet is a black ball: a technology that invariably destroys the civilisation that invents it. That’s not because we’ve been particularly careful or wise when it comes to innovation. We’ve just been lucky. But what if there’s a black ball somewhere in the urn? If scientific and technological research continues, we’ll eventually pull it out, and we won’t be able to put it back in. We can invent but we can’t un-invent. Our strategy seems to be to hope that there is no black ball. ...

"But one way to think about the possible effects of a black ball is to consider what would happen if nuclear reactions were easier. In 1933, the physicist Leo Szilard got the idea of a nuclear chain reaction. Later investigations showed that making an atomic weapon would require several kilos of plutonium or highly enriched uranium, both of which are very difficult and expensive to produce. However, imagine a counterfactual history in which Szilard realised that a nuclear bomb could be made in some easy way – over the kitchen sink, say, using a piece of glass, a metal object and a battery. ... [P]erhaps the US government would move to eliminate all glass, metal and sources of electrical current outside of a few highly guarded military depots? Such extreme measures would meet with stiff opposition. However, after mushroom clouds had risen over a few cities, public opinion would shift. Glass, batteries and magnets could be seized, and their production banned; yet pieces would remain scattered across the landscape, and eventually they would find their way into the hands of nihilists, extortionists or people who just want ‘to see what would happen’ if they set off a nuclear device. In the end, many places would be destroyed or abandoned. Possession of the proscribed materials would have to be harshly punished. Communities would be subject to strict surveillance: informant networks, security raids, indefinite detentions. We would be left to try to somehow reconstitute civilisation without electricity and other essentials that are deemed too risky. That’s the optimistic scenario. In a more pessimistic scenario, law and order would break down entirely, and societies would split into factions waging nuclear wars. ... In short: we’re lucky that making nuclear weapons turned out to be hard. We pulled out a grey ball that time. Yet with each act of invention, humanity reaches anew into the urn.

"Suppose that the urn of creativity contains at least one black ball. We call this ‘the vulnerable world hypothesis’. The intuitive idea is that there’s some level of technology at which civilisation almost certainly gets destroyed, unless quite extraordinary and historically unprecedented degrees of preventive policing and/or global governance are implemented. Our primary purpose isn’t to argue that the hypothesis is true – we regard that as an open question, though it would seem unreasonable, given the available evidence, to be confident that it’s false. Instead, the point is that the hypothesis is useful in helping us to bring to the surface important considerations about humanity’s macrostrategic situation. ... It would be bad news if the vulnerable world hypothesis were correct. In principle, however, there are several responses that could save civilisation from a technological black ball. One would be to stop pulling balls from the urn altogether, ceasing all technological development. That’s hardly realistic though; and, even if it could be done, it would be extremely costly, to the point of constituting a catastrophe in its own right. ... That leaves two options for making the world safe against the possibility that the urn contains a black ball: extremely reliable policing that could prevent any individual or small group from carrying out highly dangerous illegal actions; and two, strong global governance that could solve the most serious collective action problems, and ensure robust cooperation between states. – even when they have strong incentives to defect from agreements, or refuse to sign on in the first place. The governance gaps addressed by these measures are the two Achilles’ heels of the contemporary world order. So long as they remain unprotected, civilisation remains vulnerable to a technological black ball. Unless and until such a discovery emerges from the urn, however, it’s easy to overlook how exposed we are."
aeon.co/essays/none-of-our-technologies-has-managed-to-destroy-humanity-yet