Firms that attempted to replicate GPT with in-house data often produced shambolic “knowledge engines” that were little better than toys. These AIs supplemented the standard large-language-model training set with emails, mission statements, patent applications, legal memoranda, and other exciting selections from the internal corporate syllabus. As the buzz percolated through middle management, executives at marketing, media, and health-care firms launched ambitious initiatives, sometimes openly telling staff that many employees would be laid off once the neural nets were working. But much of what was produced was vaporware: late, expensive, and barely functional. Many users felt that AI technology simply

wasn’t ready.

Within two days, Bender had sent Gebru an outline. They later came up with a title, adding a cheeky emoji for emphasis: “On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?”

Gebru assembled a research team for the paper within Google, including her colead Mitchell. In response to the encouraging words in Dean’s annual review, she flagged the paper as an example of the work she was pursuing. “Definitely not my area of expertise,” Dean said, “but would definitely learn from reading it.”

In total, it presented four key warnings: First, large language models were growing so vast that they were generating an enormous environmental footprint, as found in Strubell’s paper.

Second, the demand for data was growing so vast that companies were scraping whatever they could find on the internet, inadvertently capturing more toxic and abusive language as well as subtler racist and sexist references.

Third, because such vast datasets were difficult to audit and scrutinize, it was extremely challenging to verify what was actually in them, making it harder to eradicate toxicity or more broadly ensure that they reflected evolving social norms and values. Finally, the model outputs were getting so good that people could easily mistake its statistically calculated outputs as language with real meaning and intent. This would make people prone not only to believing the text to be factual information but also to consider the model a competent adviser, a trustworthy confidant, and perhaps even something sentient.

The land and energy required to support these megacampuses are but two inputs in the global supply chain of data center expansion. So, too, is the extraordinary volume of minerals including copper and lithium needed to build the hardware—computers, cables, power lines, batteries, backup generators—and the extraordinary volume of potable—yes, potable—water often needed to cool the servers. (The water must be clean enough to avoid clogging pipes and bacterial growth; potable water meets that standard.) According to an estimate from researchers at the University of California, Riverside, surging AI demand could consume 1.1 trillion to 1.7 trillion gallons of fresh water globally a year by 2027, or half the water annually consumed in the UK. Those effects will not be felt evenly. Another study found that in the US, one-fifth of data centers were already drawing that water before the generative AI boom from moderately or highly stressed watersheds due to drought or other factors. And in Global South countries like Chile, it’s often the most vulnerable communities who have borne the brunt of these accelerating economies of extraction.

As more and more communities have watched data centers affect their lives, a growing number have pushed back vehemently against their unfettered development.

The bottleneck for producing novel biological weapons, for example, is not about finding a recipe, Weinstein noted. Such recipes already abound online and are easily found via Google. It is about obtaining the materials and equipment to actually make the armaments. Restricting access to so-called frontier models would thus do little to fix this. But the collateral damage of suppressing the publication of AI models would risk weakening the foundations of US AI innovation. Open source—sharing and building on code and software released to the broader community—has long been the bedrock upon which the wealth of US-based startups flourish. Restricting model weights from being published would give smaller developers fewer pathways than ever to create their own AI products and services. It would further entrench the dominance of the giants represented in the Frontier Model Forum. AI models would also become ever harder to scrutinize, such as in the work of Sasha Luccioni, Yacine Jernite, and Emma Strubell, who have relied heavily on open generative AI models to quantify the carbon and environmental costs of continuing to scale them.