Does AI Help Cyber Attackers or Defenders? Evidence from Nonpublic Vulnerabilities and Subsequent Attacks

Abstract

The release decision for frontier AI systems increasingly relies on cyber capability benchmarks, yet public vulnerability benchmarks can expose agents to previously published advisories, exploits, and fixes. The authors evaluate open-weight and proprietary AI models on exploit generation, vulnerability repair and subsequent attacks in five nonpublic software environments. Repair scores exceed attack scores in two nonpublic environments and fall below them in three; passing an initial security test is also insufficient, as another exploit succeeds in 92 of 524 non-independent defender test intervals after the initial exploit is stopped.

How to cite it

Built from the details listed on this page. Check them against the paper before you submit.

Heldt, T., Turk, M., Landolt, C., & Fritz, M. (2026). Does AI Help Cyber Attackers or Defenders? Evidence from Nonpublic Vulnerabilities and Subsequent Attacks. arXiv. https://arxiv.org/abs/2610.06584

Heldt, Tobias, et al. "Does AI Help Cyber Attackers or Defenders? Evidence from Nonpublic Vulnerabilities and Subsequent Attacks." arXiv, 2026, https://arxiv.org/abs/2610.06584.

@article{heldt2026does,
  title = {Does AI Help Cyber Attackers or Defenders? Evidence from Nonpublic Vulnerabilities and Subsequent Attacks},
  author = {Tobias Heldt and Matt Turk and Christoph Landolt and Mario Fritz},
  journal = {arXiv},
  year = {2026},
  url = {https://arxiv.org/abs/2610.06584},
}

What it is

Evaluates open-weight and proprietary AI models on exploit generation, vulnerability repair and subsequent attacks in five nonpublic software environments, using researcher-developed deterministic graders rather than LLM judges. Repair scores exceed attack scores in two environments and fall below them in three. Preprint, not yet peer-reviewed.

Topics