A new cybersecurity research dataset has identified 1,540 potentially vulnerable C and C++ functions associated with AI-assisted software development, giving researchers a large real-world collection for examining security weaknesses in code produced with artificial intelligence tools.
The dataset, called LLMVul, was developed by Mohammad Farhad and Shuvalaxmi Dass of the University of Louisiana at Lafayette. Their research paper, titled “LLMVul: A Vulnerability-Labeled Dataset of LLM-Generated C/C++ Functions from Real Production Repositories,” was submitted to arXiv on September 10, 2026.
A Dataset Built From Real Software Projects
Unlike datasets based mainly on human-written vulnerable code or controlled prompts asking AI models to generate programs, LLMVul was built from AI-assisted development activity found in public GitHub repositories.
The researchers examined 321,080 commits across 1,200 repositories and tracked AI-related provenance between November 13, 2022 and September 3, 2026.
After filtering and removing duplicate functions, the resulting dataset contained 21,430 unique C/C++ functions from 226 repositories.
The researchers identified 1,684 unique AI-attributed commits during the process. The final dataset includes code, repository and commit information, AI-tool attribution, vulnerability findings and other provenance metadata.
The project was designed to address a gap in existing research, where security datasets often rely on human-written code or artificial testing conditions rather than code that developers have actually incorporated into software projects with assistance from AI coding systems.
How AI-Assisted Code Was Identified
AI involvement was identified through GitHub commit evidence. Strong attribution signals included machine-generated co-author information associated with tools such as GitHub Copilot, Claude Code and other AI systems.
Medium-strength signals included commit messages stating that code had been generated or assisted by services such as ChatGPT, Copilot, Cursor or Claude.
Of the attributed functions, 89.2% had what the researchers classified as strong attribution evidence, while 10.8% were supported by medium-strength signals.
Claude Code Accounts for the Largest Share
Claude Code accounted for 14,221 functions, the largest share in the dataset. GitHub Copilot followed with 4,952 functions.
Claude was associated with 807 functions, Gemini with 386, Cursor with 373, ChatGPT with 298, generic LLM attribution with 242 and OpenAI Codex with 135. Devin appeared in 16 functions.
The distribution does not represent a controlled comparison of the security of individual AI coding tools. It primarily reflects the attribution signals present in the repositories included in the study.
How the Researchers Detected Vulnerabilities
The researchers used three approaches to identify security weaknesses: Semgrep, Flawfinder and a custom pattern-matching system containing 54 regular-expression patterns.
Semgrep produced 2,459 findings, Flawfinder produced 2,091 and the pattern-based system produced 1,949.
Rather than treating every individual finding as a confirmed vulnerability, the researchers used a majority-vote approach.
A function was classified as vulnerable when at least two of the three detection methods flagged it. Functions flagged by only one method were placed in an uncertain category, while those with no detections were classified as safe.
This process produced 1,540 ensemble-vulnerable functions. Another 17,211 functions were classified as safe and 2,679 were classified as uncertain.
7.2% of Functions Were Classified as Vulnerable
The 1,540 vulnerable functions represent 7.2% of all 21,430 unique functions in the dataset.
The figure should not be interpreted as saying that 7.2% of all AI-generated C/C++ code is vulnerable, because the study only covers the functions that entered the LLMVul dataset and were evaluated under its particular selection and classification process.
Memory-Safety Weaknesses Dominate the Findings
The vulnerabilities were mapped to Common Weakness Enumeration categories. The researchers report that the vulnerable functions span 17 CWE categories.
The largest detailed categories were CWE-120, buffer copy, with 494 functions, and CWE-787, out-of-bounds write, with 478.
Another 411 functions were classified under CWE-676, dangerous function use. The paper’s discussion of this category also includes findings such as divide-by-zero, null-pointer dereference, race conditions, use-after-free and untrusted-input issues that were not mapped more specifically by the analysis process.
Other reported categories included path traversal, represented by 75 functions; integer overflow, with 45; OS command injection, with 25; format-string weaknesses, with six; and weak pseudorandom-number generation, with six.
Buffer-copy and out-of-bounds-write findings together accounted for 972 functions, making memory-related weaknesses the largest part of the detailed vulnerability breakdown.
Manual Review Found Substantial Agreement
The researchers also conducted manual validation on 100 randomly selected functions from the ensemble-vulnerable population. Two reviewers independently assessed the samples, and their agreement was measured using Cohen’s kappa.
The study reports a Cohen’s kappa score of 0.79, indicating substantial agreement between the reviewers. Disagreements were subsequently discussed to reach consensus labels.
However, the manual assessment covered 100 of the 1,540 functions classified as vulnerable, rather than the complete vulnerable population.
Projects With the Largest Numbers of Vulnerable Functions
The dataset includes contributions from widely used open-source projects. Among the repositories with the largest numbers of vulnerable functions were nature-lang/nature with 227, Serial-Studio/Serial-Studio with 210, sqliteai/warp with 108, google/security-research with 77 and xroche/httrack with 72.
Other projects appearing in the ranking included ggml-org/whisper.cpp with 50 vulnerable functions, memovai/mimiclaw with 50, DarkFlippers/unleashed-firmware with 37, DavidXanatos/TaskExplorer with 29 and coturn/coturn with 28.
The researchers caution that these counts do not establish that an entire project is insecure. The figures represent functions in the LLMVul sample that met the study’s vulnerability-classification criteria.
Important Limitations
The dataset is weighted toward large public repositories because the researchers selected repositories with at least 200 stars and post-June 2022 activity.
This means the results may not reflect the security characteristics of smaller repositories, private corporate software or projects that do not leave usable AI-attribution evidence in public Git history.
AI attribution is one of the central limitations identified by the researchers. Developers may use an AI assistant without leaving an attribution signal in Git metadata, which means such code may never enter the dataset.
The problem can also arise when a developer substantially rewrites AI-generated code before committing it. In that situation, determining how much of the final implementation came from an AI system becomes difficult.
Why Function-Level Analysis Matters
Another limitation comes from the function-level analysis itself. A function that appears unsafe in isolation may be protected by validation elsewhere in the application, while a function that appears safe may become vulnerable because of how another component calls it.
The researchers note that this context problem can contribute to both false positives and false negatives.
They also explain that CodeQL was not used because its C/C++ extraction process requires compilable translation units, which does not fit the isolated-function structure of the dataset.
What the Dataset Does — and Does Not — Show
LLMVul does not establish that one AI coding assistant produces less secure code than another.
The number of functions attributed to each tool is highly uneven, and the study was not designed as a controlled test of competing systems.
It also does not establish that AI caused every vulnerability identified in the dataset or that every flagged function is exploitable in its original software context.
Instead, the research provides a large collection of C/C++ functions associated with AI-assisted development, together with vulnerability classifications and information about their provenance.
Future Research
The researchers say future work could expand the approach to languages including Python, Java and Rust.
They also propose automated build infrastructure that would allow additional security-analysis tools to operate against compilable code and provide more context for vulnerability assessment.
Such work could help researchers study AI-assisted software security across different programming languages and development environments.
Dataset and Research Materials
The LLMVul dataset is publicly available through Zenodo, while the researchers have also published the project materials and code through their GitHub repository.
The dataset contains 34 fields covering information such as repositories, commits, source files, function bodies, programming language, AI tools, attribution strength, vulnerability labels and the results of the three detection approaches.
The project is released under a CC BY 4.0 license, while the accompanying scripts are released under an MIT license.
Researchers and Funding
LLMVul was developed by Mohammad Farhad and Shuvalaxmi Dass of the School of Computing & Informatics at the University of Louisiana at Lafayette.
Dass is an assistant professor whose listed research interests include software security, machine learning, ethical hacking and cybersecurity education.
The research was supported in part by the U.S. National Science Foundation under Grant OIA-2437963 and the Louisiana Board of Regents.
Why LLMVul Matters
LLMVul does not demonstrate that AI-generated software is inherently insecure.
Its significance is that it provides researchers with a large, provenance-aware sample of C and C++ functions associated with AI-assisted development in real public repositories, alongside security classifications that can be studied and independently evaluated.
The findings also show that the security problems detected in the dataset are largely familiar C/C++ weaknesses, with buffer-copy and out-of-bounds-write findings accounting for the largest share of the detailed vulnerability classifications.
Discover more from Aree Blog
Subscribe now to keep reading and get access to the full archive.
