The bottleneck for producing novel biological weapons, for example, is not about finding a recipe, Weinstein noted. Such recipes already abound online and are easily found via Google. It is about obtaining the materials and equipment to actually make the armaments. Restricting access to so-called frontier models would thus do little to fix this. But the collateral damage of suppressing the publication of AI models would risk weakening the foundations of US AI innovation. Open source—sharing and building on code and software released to the broader community—has long been the bedrock upon which the wealth of US-based startups flourish. Restricting model weights from being published would give smaller developers fewer pathways than ever to create their own AI products and services. It would further entrench the dominance of the giants represented in the Frontier Model Forum. AI models would also become ever harder to scrutinize, such as in the work of Sasha Luccioni, Yacine Jernite, and Emma Strubell, who have relied heavily on open generative AI models to quantify the carbon and environmental costs of continuing to scale them.

This is where Te Hiku’s story diverges completely from OpenAI’s and Silicon Valley’s model of AI development. Intimately familiar with the devastating effects of colonial dispossession, Jones and Mahelona were determined to carry out the project only if they could guarantee three things—consent, reciprocity, and the Māori people’s sovereignty—at every stage of development. This meant that even before embarking on the project, they would get permission from the Māori community and their elders, asking them if the endeavor was even something they wanted; to collect the training data, they would seek contributions only from people who fully understood what the data would be used for and were willing to participate; to maximize the model’s benefit, they would listen to the community for what kinds of language-learning resources would be most helpful; and once they had the resources, they would also buy their own on-site Nvidia GPUs and servers to train their models without a dependency on any tech giant’s cloud.

The critiques that I lay out in this book of OpenAI’s and Silicon Valley’s broader vision are not by any means meant to dismiss AI in its entirety. What I reject is the dangerous notion that broad benefit from AI can only be derived from—indeed, will ever emerge from—a vision for the technology that requires the complete capitulation of our privacy, our agency, and our worth, including the value of our labor and art, toward an ultimately imperial centralization project.

Te Hiku shows us another way. It imagines how AI and its development could be exactly the opposite. Models can be small and task specific, their training data contained and knowable, ridding the incentives for widespread exploitative and psychologically harmful labor practices and the all-consuming extractivism of producing and running massive supercomputers. The creation of AI can be community driven, consensual, respectful of local context and history; its application can uplift and strengthen marginalized communities; its governance can be inclusive and democratic.

In her 2019 talk at NeurIPS, during the Queer in AI workshop, Ria Kalluri, an AI researcher at Stanford, proposed an incisive alternative to the question of how to ensure AI does “good.” Goodness, benefit to humanity—these terms will always be in the eye of the beholder. Rather, we should ask how AI shifts power: Does it consolidate or redistribute that power? To put it in the frame of this book, does it continue to fortify the empire, or does it begin to wrest us back toward democracy?

Redistributing knowledge will also need policies that require companies to relinquish key details about the training data and technical specifications of their models and supercomputers. Only then could independent corporate model evaluators do their work. UC Berkeley researcher Deborah Raji, who has continued to engage with global policymakers after the Schumer forums, says this is also a bare minimum for guaranteeing the real-world safety of corporate systems. That is, not the theoretical rogue AI harms of Doomerism, but the existing real-world harms, from discrimination to misinformation to job automation, that consumers and communities can already face if widely deployed models aren’t properly tested. “We have the CFPB that monitors consumer finance products. We have the FDA that monitors medical devices. But for some reason when it comes to AI products, there’s just no oversight,” Raji says. AI models should in fact require more transparency than the average product. “These are data-defined systems. They’re not deterministic. So we need to know more about these systems to understand what they’re doing.” Such transparency is additionally crucial for measuring the impact of AI on the environment. In this regard other products once again already submit to evaluations that AI products do not. “If you’re using a car, if you are buying an appliance, you have an Energy Star rating,” says Sasha Luccioni at Hugging Face. “But AI is so integrated into our society, so widely used in products, and we don’t have any information about the sustainability of these systems.”