AI Security Institute

An important step towards tackling this serious risk. We’re pleased to have supported the development of these plans and will continue working with partners to ensure the right safeguards are in place to protect children from harm.

Last week, we hosted our inaugural Alignment Conference, in partnership with FAR.AI . The event bought together an interdisciplinary delegation of leading researchers, funders, and policymakers to discuss urgent open problems in AI alignment. Ensuring that future AI systems act as we intend will require a rapid, cross-disciplinary expansion of the AI alignment field. Progress hinges on contributions from fields spanning cognitive sciences to learning theory. Our conference deepened this technical collaboration through five research tracks: 1️⃣ Theoretical Computer Science  2️⃣  Learning Theory & Learning Dynamics  3️⃣ Economic Theory  4️⃣  Cognitive Science & Scalable Oversight + Evaluations  5️⃣ Explainability  Learn more about AISI’s work to accelerate research in AI alignment: https://orlo.uk/KNNQK Read our research agenda: https://orlo.uk/sdJFg

We collaborated with Lakera to design the backbone breaker benchmark (b³) – a new open-source evaluation for LLM agents. The b³ is built on more than 194,000 crowdsourced adversarial attacks and uses ‘threat snapshots’ to identify vulnerabilities without modelling full workflows. Read the full paper: https://lnkd.in/dy-bt-uC

📢 Announced today: Adam Beaumont has been appointed AISI’s Interim Director. Adam joins us from GCHQ, where he served as Chief AI Officer. His experience spans tackling security challenges through public-private partnerships, leading an AI research lab, and advising on security and defence policy in both Whitehall and Washington. His work will see him build connections across the UK government, international community, broader AI ecosystem to advance AISI’s mission of ensuring AI serves the public good. https://orlo.uk/1fWns

Several AI developers aim to build systems that match or surpass humans across most cognitive tasks. Today’s AI still falls short. Among AISI’s priorities is to forecast the development of AI much more powerful than today’s. Such systems could be extremely beneficial – but may also pose national security risks or cause labour market disruption. In a new report, we draw on expert interviews to identify four categories of limitation that still constrain today’s AI systems: 1️⃣ Task-specific limitations: Performance on long tasks, in complex environments, and on tasks that are hard to verify 2️⃣ Reliability: Reducing error rates and improving meta-awareness 3️⃣ Adaptability: Working with local context and continual learning 4️⃣ Original insight: Novel contributions of scientific value For each category we ask: where do existing systems still struggle? What would we expect to see if these obstacles were overcome? In so doing, we hope to provide tools for the AI safety and national security communities to monitor and forecast AI capabilities. The trajectory of AI development is highly uncertain, and unforeseen bottlenecks could emerge. AISI will continue to gather evidence on this trajectory as capabilities advance. Learn more in our blog: https://lnkd.in/e5uJ2g2r Read the full report: https://lnkd.in/eKrwVUwr

How can we prevent harm from AI systems that pursue unintended goals? AI control is a promising research agenda that seeks to address this critical question. Today, we’re excited to launch ControlArena – our library for running secure and reproducible AI control experiments. The typical AI control experiment is complex to design and run, often requiring custom infrastructure to be built from scratch. ControlArena streamlines this process, by providing a suite of pre-built evaluation environments behind a simple interface. Researchers can easily define each element of a control experiment: ➡️The setting in which the agent operates ➡️Legitimate work (“main tasks”) vs undesirable objectives (“side tasks”) ➡️Control protocols to prevent the success at side tasks ControlArena also provides all the tools you need to load evaluation logs, analyse results, and generate customisable visualisations. We hope that ControlArena will cut development time, make results easier to reproduce, and lower the bar to entry for AI control research across the AI safety, cybersecurity, and machine learning communities. Read more on our blog: https://lnkd.in/eGtzAwuP Get started with ControlArena: https://lnkd.in/eR8iumsb

Measuring how often an AI agent succeeds at a task can help us assess its capabilities – but it doesn’t tell the whole story. We’ve been experimenting with transcript analysis to better understand not just how often agents succeed, but why they fail. Our model evaluations generate thousands of transcripts, which can contain an entire novel’s worth of text. They are a record of everything the model did during a task, including the external tools it accessed, and its outputs at each step. In a recent case study, we analysed almost 6,400 transcripts from AISI evaluations of nine models on 71 cyber tasks. We studied several features of these transcripts, including overall length and composition, and the agent’s commentary throughout. We found that there are many reasons a model may fail to complete a task, beyond capability limitations. These can include safety refusals, lack of compliance with scaffolding instructions, or difficulty using tools. We’re sharing our analysis to encourage others conducting safety evaluations to review their own transcripts, in a systematic and quantitative way.  This can help foster more accurate and robust claims about agent capabilities. Read more on our blog: https://lnkd.in/eiCn6zkP

Alongside Anthropic and the The Alan Turing Institute, we’ve conducted the largest investigation of data poisoning to date. Data poisoning occurs when individuals distribute online content designed to corrupt an AI model’s training data, potentially producing dangerous behaviours, including the insertion of backdoors – specific phrases that trigger an otherwise-hidden behaviour. Backdoors can be used to degrade system performance or even make models perform harmful actions like exfiltrating data. We found that as little as 250 malicious documents can be used to “poison” a language model, even as model size and training data grow. Previous work had assumed that attackers would need to poison a certain percentage of data to succeed, but our results suggest that this is not the case. This means that poisoning attacks could be more feasible that previously believed. We are releasing this content to raise awareness of these risks and spur others to take defensive action to protect their models. We’ll be continuing our work with Anthropic and other frontier developers to strengthen model safeguards as capabilities improve. Learn more on our blog: https://lnkd.in/egDzXC_g Read the full paper: https://lnkd.in/eP47Dfse

Our recent large-scale study investigated how often people use AI to research political issues, and whether it increases belief in misinformation. ➡️ Read the key takeaways in our new blog: https://lnkd.in/eAwbgVpZ

We are excited to advance our partnership with the US Center for AI Standards and Innovation (CAISI) through the UK-US tech partnership, including collaborating on best practice for advanced AI model security.