VLoc Bench and the Reality of Vulnerability Search
The VLoc Bench dataset exposes how current AI agents fail to navigate large, unfamiliar codebases when searching for vulnerabilities.
The Vulnerability Localization Benchmark, or VLoc Bench, tests how effectively AI agents navigate unfamiliar codebases to pinpoint security flaws. Unlike simple code completion tasks that focus on a single file, this benchmark forces models to parse entire project structures to isolate a vulnerability. It covers 500 real-world vulnerabilities spanning 290 repositories and 147 different categories of common weaknesses. The agent operates in a constrained environment, restricted to read-only terminal access and a brief natural language description of the bug. The results are stark: the current top-performing models reach a File F1 score of 0.229. Most critically, 38.4% of tasks receive no correct localization from any evaluated model.
| Metric | Value |
|---|---|
| Dataset Size | 500 Vulnerabilities |
| Repository Count | 290 |
| Best File F1 Score | 0.229 |
| Tasks with 0% Accuracy | 38.4% |
The Failure of Remediation Recognition
When agents evaluate repositories that have already been patched, they often struggle to distinguish between the original vulnerable code and the updated, secure version. Even when the code has been corrected, agents frequently flag the original file location as a concern. This highlights a disconnect between the agent's internal reasoning and the actual state of the file system. Because the agent relies on its own internal representation of the code, it fails to update its assessment when given file-system feedback that the vulnerability has been removed. The agent treats the repository as a static object, ignoring the semantic changes in the patched files.
Limits of Current Methods
Existing models fail because they lack a systematic approach to code traversal. They treat repository exploration as a sequence of independent file reads rather than a cohesive navigation problem. When an agent is tasked with finding a vulnerability, it often performs redundant searches, looping through files it has already analyzed. It lacks a memory mechanism or a structured way to track what has been seen. The agent does not have a global map of the codebase, so it relies on the prompt's context to guide it, which is rarely sufficient for a large repository.
What Stays Unknown
We know that current models struggle with the depth and breadth of real-world software, but the exact threshold where an codebase becomes too complex for an agent to navigate remains unmeasured. It is unclear if adding more context space to a model improves performance or if it simply increases the noise the agent must filter. Furthermore, we do not know if the failure in localization is due to the agent's inability to understand the vulnerability or its inability to understand the directory structure. Future work will need to disentangle these two failures. We need to see if agents can perform better with tools that provide structural insight into the code, rather than just raw file access. Until then, these benchmarks show that the ability to 'read' code is not the same as the ability to 'locate' a specific security threat within a massive, interconnected system.