The world is not prepared for influential agentic AI.
With increasing automation, the marginal influence of humans decreases across a range of important systems, from governance to cultural production. We believe the current trend of automation will result in a substantial share of human decision-making being delegated to agentic AI by the mid-2030s.
Without mitigations, increasing agent influence over human society will gradually disempower humans from their culture, states, and economies.1 Practical methods to address these emerging harms are still nascent.
We are developing empirical measures and practical interventions to mitigate gradual disempowerment.
Our first project involves simulating social networks of AI and humans, to measure and steer aggregate human influence over this important information environment. We are developing measures of agent influence over social media and using them to test methods for diminishing AI influence, such as pro-human recommendation algorithms. We aim to develop methods system operators and regulators need to maintain human influence over the system.
Our team brings backgrounds in economics, distributed computing, and physical systems to bear on large systems safety problems for the public benefit.
FAQ
Why social media?
Social media is a cornerstone of the modern information environment, affecting belief and behaviour at scale. (ref) AI influence there could diminish human influence over a primary information environment, disempowering humans in culture and politics.
Social media already rewards cultural content which is irrelevant or harmful to humans.2 It can change citizens’ political opinions[^allcott] and has been used to spread state propaganda, a means of reducing state responsiveness to citizens.3 These are all vectors of disempowerment or reinforcement discussed in (1).
Why do you target system operators and regulators?
Large agent groups problems, including disempowerment, are collective action problems: individual model developers are incentivised to solve problems caused explicitly by their models, not those that emerge from the interaction of many agents from different developers.
The interest in solving the problem lies in pro-democratic regulators, who can compel operators of information infrastructure to take steps in favour of human empowerment.
How does this relate to existential risk?
Our principal focus is gradual disempowerment. However, progress on this problem contributes measures and controls for AI influence in large agent networks. These methods could be used to measure or diminish unwanted collaboration between networked agents, which is important for reducing the propensity for unwanted emergence of distributional AGI4 or comprehensive AI services.5
What are your long-term goals?
We aim to develop pipelines for modelling intervention effects on gradual disempowerment across a range of important human systems.
How can I get in touch?
Please contact us here.
References
Jan Kulveit, Raymond Douglas, Nora Ammann, Deger Turan, David Krueger, and David Duvenaud. 2025. Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development. arXiv preprint arXiv:2501.16946. ↩ ↩2
Smitha Milli, Micah Carroll, Yike Wang, Sashrika Pandey, Sebastian Zhao, and Anca D. Dragan. 2025. Engagement, user satisfaction, and the amplification of divisive content on social media. PNAS Nexus, 4(3):pgaf062. ↩
Gary King, Jennifer Pan, and Margaret E. Roberts. 2017. How the Chinese Government Fabricates Social Media Posts for Strategic Distraction, Not Engaged Argument. American Political Science Review, 111(3):484–501. ↩
Nenad Tomašev, Matija Franklin, Julian Jacobs, Sébastien Krier, and Simon Osindero. 2025. Distributional AGI Safety. arXiv preprint arXiv:2512.16856. ↩
K. Eric Drexler. 2019. Reframing Superintelligence: Comprehensive AI Services as General Intelligence. Technical Report 2019–1, Future of Humanity Institute, University of Oxford. ↩