In recent years, the spectre of the Manhattan Project was often raised as an example for how innovation and technology can and should be done in the US. In 2024, as a U.S. congressional commission (of interested parties) suggested, we need a “Manhattan Project Style AI initiative”.
Rather than being a cautionary tale of how certain technological pursuits might produce grave and possibly irresponsible consequences, the Manhattan Project is – in this context – considered as a blueprint for action, an unambiguous good which ought to be repeated.
It’s a strange revisionism, but one that should perhaps prompt us to look back into a time period in which the sharp drift toward autonomy first came underway. At that time, in the 1940s and 50s, some, if not many, mathematicians and scientists, philosophers and theorists already expressed many of the reservations about the new human-machine constellation we have today, and perhaps they were able to still see the thorniest of the problems that would arise when humans and machines come to merge in the form we take for granted today. This might help us understand some of the logics and structures of the wider constellation that make up the materials and practices of contemporary warfare and crystallise how the circulation of certain technological logics function as a social and political ordering principle.
Consider, for example, one of the key figures in the history of cybernetics and AI, Norbert Wiener. Wiener played an important role in advancing automation and autonomy in weapons systems, but on the heels of the Manhattan Project and the highly controversial use of the nuclear bomb he, and others, became increasingly wary about the role scientists and mathematicians came to play in the production of technologies for war; scientists. “who have been thrown most violently off stride, who have left academic pursuits for the making of strange destructive gadgets”, as the then Director of the Office of Scientific Research and Development, Vannevar Bush publicly noted.
Unlike Bush, however, Wiener was ill at ease with the increasingly prevalent perspective which viewed technology “not so much as applied science, but rather as applied social and moral philosophy”. As a primary logic for how we understand the world and do things in it. In other words, Wiener saw what seems less obvious to us now, that the logic of computation and information technology has become a dominant mode of (philosophical) reasoning and might come to shape how we think and act in the world. Wiener was not alone in this realization: Herbert Marcuse, Lewis Mumford, Hans Jonas, Günther Anders, Hannah Arendt, Jacques Ellul and others made similar observations about technology and modern society in those early years.
The unbridled development of AI military systems
Today, the development and deployment of increasingly autonomous AI-enabled systems, including those that make or recommend military targeting decisions, is advancing nearly unimpeded. We have seen many reports come out on the highly destructive consequences of AI decision support systems used in Gaza.
And in particular the Russia-Ukraine conflict has tragically given an uplift to the perceived need to develop fully autonomous weapons swiftly and without much constraint, while the conflict itself has been producing extremely valuable data with which AI-weapon developers are hoping to refine their systems. Active conflicts become an open laboratory for testing, trialling and iteration.
Regardless of any discussions about the promises of accuracy and precision and sophistication of AI-enabled weapons, fact is that until now systems that are supposed to be used in open complex battle contexts are not likely to be very robust or reliable, nor is there any safeguard that they will be calibrated or used in a way that restrains violence. Quite the contrary seems likely.
Despite all this, the promises made about military AI systems are fantastical. Claims that AI will help win wars swiftly, that war is just a software problem, that AI can make the globe transparent and so on, are a staple in this realm and they fuel projects like the very ambitious Joint All Domain Command and Control programme (JADC2), a US State department project to create an AI powered network of all defense branches.
The JADC2 vision aims for seamless data connectivity to enable real-time action at speed and scale across all domains—from the ocean floor to outer space, including air, land, sea, space, and cyber.
The analogy that is often used by the US Department of Defense for how this vision should work is the ride-sharing platform Uber. Instead of a ride, you coordinate rockets, anywhere, everywhere.
The flaws of machines learning
What is often overlooked in these ambitious programmes is that weapons that employ AI for the full kill-chain are likely to be hampered by incomplete, low-quality, incorrect or discrepant data. This, in turn, will lead to highly brittle systems and biased, harmful outcomes. Conflict environments are messy, dynamic and, importantly, adversarial. There is a ton of variability for which a system is not trained or verified.
Moreover, data for AI is often mislabelled, and research in machine learning for AI is by far not as scientifically robust as it might appear from media accounts. This condition will inevitably lead to faulty decisions, to misapplications of force and other harmful errors – harmful to those in the crosshairs of these errors.
As I have argued elsewhere in more detail, where techno-logics dominate, harmful outcomes become simply folded into the technological practice either as accidents or part of the process, as the excess of innovation.
Machine learning, as a technique, is underwritten by a logic of waste, or excess, if you will. Of doing, failing, trying again, getting closer, failing again … and so on and using all necessary resources to do so in the hopes of approaching some idea of perfection. Much like the current iteration of Silicon Valley as an industry itself.
The venture capital logic to technology
This logic of machines is worth paying attention to. They are not conceived in, nor exist in a vacuum, or per se. They are always social as well as technical. Felix Guattari has said that machines are hyper-concentrated forms of certain aspects of human subjectivity.
And we might want to look to the technology industry to better understand our present condition, given that Silicon Valley, and its financial backers, appear to be increasingly in the driving seat – at least in the US – when it comes to dictating the rhythm and pace of military technological innovation, and indeed the cultural shift toward full autonomy, or at the very least toward marginalizing the human-as-human.
There is a highly complex and firmly entwined relationship between Silicon Valley, ideas of military progress and venture capital aims that goes back many decades. The venture capital logic embraces a similar valorisation of error and iteration as the AI logic does: investing is high-risk high-reward, some ventures work out, many do not. Up to 80% of investments might fail, but the home runs will make up for all the losses. It’s always a bet.
The dynamics of this relationship and the increasingly notable dominance of private sector rhetoric, and practices, for military operations, should, however, give us pause for thought. What happens when the military picks up the pace of the technology industry and its wider ethos of do-first-ask-for forgiveness-later?
The parameters of the tech industry are built on speed, scale and competition. Facebook – now Meta – has built its business around the ethos that there is intrinsic value in getting products to market at speed, regardless of whether these technological ideas are fully fit for use.
The iconic ‘Move fast and break things’ motto remains fundamental in illustrating this ethos. Another one of then-Facebook’s slogans was ”Done is better than perfect,”. This is the essence of the business practice – the Hacker Way: the technology sector must move fast, it is about winning the race to getting a product to market, to getting a contract, to “blitzscaling”, to creating a monopoly from which to extract monopoly rents – that means making extraordinary amounts of money from ultimately not giving customers a choice. It locks the customer into a relationship of dependency and reliance from which it becomes too costly to extricate oneself with ease. From this position, the vendor can extract continuous monopoly rents.
Only by embracing the logic of error and mistakes can those involved move fast enough to gain an advantage over competitors. This is precisely why the military AI industry cultivates the framing of a competition, of the AI arms race. It needs it to make the logic work.
It is also increasingly an attitude that is being fostered for the defence sector, advocated for by those that need this framing. At the US Armed Services Committee hearing, Palantir’s Chief Technology Office gave evidence advocating for “more crazy” and for “letting chaos reign” in the military acquisition and procurement process, so that the necessary incentives can be fostered. Regulatory limitations, he thinks, are too constrictive, and he “would gladly accept more failure if it meant that we had more catastrophic success”. What kind of success this might be, or what the implications are for failure, remains unaddressed, but it is clear that Palantir’s CTO speaks with a venture capital logic in mind.
Shortcuts and post hoc fixes are always justified with the understanding that mistakes can always be addressed later – through updates, and iterative adjustments and so on. The mandate to embrace mistakes and errors is valorised, speed trumps everything. The potential waste produced in this practice is made up for by the promise of high financial rewards.
Overpromising is a feature of this ethos and it relies on the now firmly entrenched idea that, at some point in the future, AI is going to be so good, so perfect – not yet, but it will be, the assumption is – that success, too, will be inevitable. A future with AI is inevitable for success. So, do it fast and big. Speed and scale. And this is precisely what underpins the problematic AI hype cycle – military or otherwise.
Silicon Valley’s DNA into Pentagon
It is worth taking a closer look at those that make the technologies that are hyped the most, the loudest. Not because these will necessarily be the technologies that will stand the test of time, but rather because they will shape what is being paid attention to, and how ideas of war and victory are framed.
The military AI market is an enormously lucrative and promising market currently valued at USD 9.2 billion, and aggressive newcomers have been moving in swiftly, backed by extraordinary amounts of venture capital, often from a handful of powerful venture capital players associated with prominent venture capital funds like Andreessen Horowitz and Founders Fund.
Start-up unicorns such as Anduril, Palantir, Shield AI and many others are all invested in the billions to become significant players in this market. It is those types of players that – although they all profess to be deeply concerned about the legality and ethics of their products, are actually not displaying particular concerns about either of them. Palmer Lucky, CEO of Anduril, himself quips that even if he’s sick in the head, “you need people like me, who are sick in that way and who don’t lose any sleep making tools of violence.” Alex Karp CEO of Palantir meanwhile publicly says he believes in “inflicting pain” and advocates for a U.S. foreign policy that should focus on “inflicting pain for generations, on anyone who hurts an American”
To be clear, technological innovations and military endeavours have long gone hand in hand. Traditionally, it was the requirements of military organisations that would set the pace and structure for technological innovations, and defence companies had, previously, worked in tandem with the military to foster innovation and ensure competitive advantage. This has now changed. New defence companies see themselves as product providers rather than government contractors and this means that the need for an already-created-product must be fostered in the existing defence market.
The former CEO of Google, and defence advisor to previous administrations, Eric Schmidt has been a very vocal advocate of making military processes and procedures much faster and he has forged very close ties with Washington to promote his “Silicon Valley worldview where advances in software and A.I. are the keys to figuring out almost any issue.”
And others, including Peter Thiel, have worked hard over the past decade to install software-friendly policies into governments through forging ties with the government. For Schmidt and others like him, traditional processes and procurement procedures in military operations are far too slow and he appears determined to align the timeframes of action between the military world and the technology world, in a push to modernize the US military- to make it like Silicon Valley. What Schmidt, in particular, often fails to acknowledge is that he has various investments, venture capital and otherwise, in firms that need this defense overhaul to be successful.
The success of Schmidt’s financial investment in military AI (and that of other venture capital investors) hinges precisely on refashioning all social, political and in this instance military processes in the image of the technology and in the image of the technology sector.
For the investment to bear fruit, those with vested interests need the US government to operate at machine speed, not human speed – in all aspects. Venture capital interests don’t have time to wait around for multi-year contract processes, they need quick turnaround, so that startups can get contracts, raise more capital and grow in valuation. The product that venture capital companies offer to their clients, that is, those that invest in their funds, is not the technology, nor the company as such, it is simply growth in valuation, growth capital gains. For this to work, the environment must be molded accordingly.
Recall Wiener’s insight that where certain technologies become more pervasive, so does the logic of their inner workings. As AI and digital technologies are dissolved into our social, political and military institutions the structural and logical signature of the technology becomes a substrate, an infrastructure, which mediates and directs ideas and actions along its functional lines. Including normative ideas. Within the logic of AI computation, normative judgments about desirable actions and practices are rendered through the lens of technological processes and business practices. It is not the human that defines horizons of possibility, it is the computer-as-ethic and more importantly, its industry.
What is worrying about this constellation – Silicon Valley’s active interest in infusing its DNA into the Pentagon – is that the technological process it foregrounds is often paired with a crude worldview of the worthy defeating the dark forces. Of good and evil, of constant battle, of urgency and existential fear, contrasted by some equally crudely drawn utopia of victory, of a glorious ‘after’.
Not unlike the plot in the novels from which two of the more prominent companies draw their name – Anduril and Palantir are both references to the Lord of the Rings. And these increasingly vocal and extremely well networked and influential set of actors offering such views are presenting themselves not merely as businessmen, but as saviors, as prophets and as politicians who alone are able to restore (The West, the world, the US, civilization) to some former glory. These are powerful narratives, and a terrifying outlook which not only aims to enroll us all into the business of war, just like the Manhattan Project did for the U.S., but it also distorts the very horrendous realities of war and its far reaching consequences for our contemporary era.
