White Paper: Cyforta Agentic Scanning on the OWASP Benchmark

Abstract

Static Application Security Testing (SAST) has long suffered from high false-positive rates and developer alert fatigue. This paper evaluates Cyforta 1.0, an agentic LLM vulnerability scanner that combines semantic context reasoning with automated Proof-of-Concept (PoC) exploit verification. Evaluated against all 2,740 test cases of the OWASP Benchmark (v1.2), Cyforta 1.0 achieves an industry-leading Youden’s Index of 0.840 (97.3% Sensitivity, 13.36% False Positive Rate)—establishing a new standard for autonomous, high-precision AppSec in modern CI/CD pipelines.

Executive Summary #

Modern software engineering requires security tooling that operates at the speed of continuous deployment without sacrificing accuracy. Traditional commercial SAST tools struggle on standardized benchmarks, often yielding high false-positive rates that create severe alert fatigue and necessitate time-consuming manual triage.

The Cyforta 1.0 Agentic Scanning Framework represents a departure from static syntax and pattern matching. By orchestrating specialized LLM agents through a pipeline of taint-guided reasoning and adversarial review, Cyforta 1.0 understands complex code intent and custom sanitization logic that baffles traditional analyzers.

Evaluated across the entire OWASP Benchmark v1.2 suite (2,740 test cases), Cyforta 1.0 achieved 97.3% Sensitivity (catching 1,377 of 1,415 true vulnerabilities) and an 86.6% Specificity rate, resulting in a Youden’s Index of 0.840 and an F1-score of 92.76%. This high diagnostic fidelity enables engineering teams to deploy automated security gates in CI/CD pipelines where findings are trusted to block high-risk regressions without human bottlenecking.

Architecture & Core Technologies #

Cyforta 1.0 operates through a five-stage agentic pipeline designed to deliver comprehensive vulnerability recall while aggressively suppressing false alarms:

STAGE 01

Semantic Context Parsing

Ingests the target codebase to construct a semantic representation of data sources, transformation layers, and sinks, moving beyond rigid abstract syntax trees to understand overarching program intent.

STAGE 02

Taint-Guided LLM Reasoning

Performs virtual data-flow analysis by tracking how untrusted input propagates across functions and custom sanitization routines, uncovering non-obvious logic vulnerabilities.

STAGE 03

Adversarial LLM Review

Subjects candidate findings to an adversarial reviewer agent that challenges reachability, identifies framework-level protections, and filters out non-exploitable theoretical findings.

STAGE 04

Deterministic Validation

Cross-references model reasoning with AST verification to ensure cited source paths, symbols, and parameters strictly exist in the codebase, preventing hallucinations.

STAGE 05

Dynamic PoC Verification

Synthesizes unit-level Proof-of-Concept exploit test cases and executes them in an isolated sandbox. Only vulnerabilities that trigger confirmed exploit conditions are elevated to blocking status.

OWASP Benchmark Overview & Methodology #

The OWASP Benchmark is an open-source evaluation suite designed to assess the accuracy, speed, and coverage of automated software vulnerability scanners.

Test Suite Composition

Version 1.2 consists of 2,740 total test cases across 11 vulnerability categories:

True Vulnerabilities (1,415 cases): Test cases containing actual, exploitable vulnerabilities across data-flow sinks.
Safe Controls & Decoys (1,325 cases): Code paths containing effective validation, encoding, or framework defenses where vulnerability alerts constitute false positives.

Evaluation Formulae

Performance is quantified using standard diagnostic classification metrics:

True Positive Rate (Sensitivity / TPR): TPR = TP / (TP + FN) — measures vulnerability discovery coverage.
True Negative Rate (Specificity / TNR): TNR = TN / (TN + FP) — measures precision on safe code.
False Positive Rate (FPR): FPR = FP / (TN + FP) = 1 - Specificity.
Youden’s Index (J): J = Sensitivity + Specificity - 1 — summarizes overall diagnostic power, where 1.0 represents perfect classification and 0.0 represents random guessing.

Comparative Analysis #

Cyforta 1.0 was evaluated against benchmark averages for traditional commercial SAST, open-source linters, rule engines, and foundation LLMs on the OWASP Benchmark v1.2:

Industry Accuracy Comparison (OWASP Benchmark v1.2) 2,740 Test Cases

Diagnostic accuracy benchmark results comparing Cyforta 1.0 against traditional commercial SAST, linters, rules-based engines, and foundation LLMs.
Tool Category	Youden's Index (J)	Sensitivity (TPR)	False Positive Rate (FPR)
Cyforta 1.0 Agentic Scanner	0.840	97.30%	13.36%
Traditional Commercial SAST	0.26 – 0.33	45.0% – 65.0%	25.0% – 40.0%
Open-Source Linters	0.15 – 0.35	30.0% – 55.0%	15.0% – 30.0%
Rules-Based SAST	0.30 – 0.40	50.0% – 70.0%	20.0% – 35.0%
Raw Foundation LLMs	0.10 – 0.25	40.0% – 60.0%	35.0% – 50.0%
Hybrid Research Frameworks	0.45 – 0.55	70.0% – 85.0%	15.0% – 25.0%

Performance Summary (OWASP) #

Evaluation results for Cyforta 1.0 across all 2,740 test cases (1,415 vulnerable cases and 1,325 safe controls):

Cyforta 1.0 Empirical Confusion Matrix & Metrics OWASP Benchmark v1.2 Full Suite

Detailed breakdown of total evaluated test cases, confusion matrix, sensitivity, specificity, precision, F1-score, and Youden index for Cyforta 1.0.
Metric	Score / Count	Definition & Context
Total Test Cases	2,740 / 2,740	100.0% coverage (1,415 true vulns, 1,325 safe controls)
True Positives (TP)	1,377	Correctly identified vulnerabilities
False Positives (FP)	177	Safe test cases incorrectly flagged
True Negatives (TN)	1,148	Correctly recognized safe controls
False Negatives (FN)	38	Missed vulnerabilities (2.7% miss rate)
Sensitivity (Recall / TPR)	97.30%	TP / (TP + FN)
Specificity (TNR)	86.60%	TN / (TN + FP)
False Positive Rate (FPR)	13.36%	FP / (TN + FP)
Precision (PPV)	88.61%	TP / (TP + FP)
F1-Score	92.76%	Harmonic mean of Precision and Sensitivity
Youden’s Index (J)	0.840	Overall diagnostic capability (Sensitivity + Specificity - 1)

Category Breakdown #

Performance breakdown across all 11 vulnerability categories evaluated in OWASP Benchmark v1.2:

Performance by CWE Vulnerability Class 11 Test Categories

Performance breakdown of Cyforta 1.0 across all 11 OWASP Benchmark test categories showing test counts, sensitivity, specificity, and Youden index.
Category	Total	Vulns	Safe	TP	FP	TN	FN	Sensitivity	Specificity	Youden J
Weak Randomness	493	218	275	217	10	265	1	99.5%	96.4%	+0.959
XPath Injection	35	15	20	15	1	19	0	100.0%	95.0%	+0.950
Path Traversal	268	133	135	133	10	125	0	100.0%	92.6%	+0.926
Secure Cookie Flag	67	36	31	36	3	28	0	100.0%	90.3%	+0.903
SQL Injection	504	272	232	272	27	205	0	100.0%	88.4%	+0.884
Cross-Site Scripting (XSS)	455	246	209	246	25	184	0	100.0%	88.0%	+0.880
Trust Boundary	126	83	43	83	6	37	0	100.0%	86.0%	+0.860
LDAP Injection	59	27	32	27	5	27	0	100.0%	84.4%	+0.844
Command Injection	251	126	125	114	15	110	12	90.5%	88.0%	+0.785
Weak Cryptography	246	130	116	130	48	68	0	100.0%	58.6%	+0.586
Weak Hash	236	129	107	104	27	80	25	80.6%	74.8%	+0.554
TOTAL	2,740	1,415	1,325	1,377	177	1,148	38	97.3%	86.6%	+0.840

Key Observations & Strengths #

100% Detection in 8 of 11 Vulnerability Classes: Cyforta 1.0 achieved complete recall across SQL Injection, Cross-Site Scripting (XSS), Path Traversal, Trust Boundary Violations, LDAP Injection, XPath Injection, Secure Cookie Flags, and Weak Cryptography.
Low 2.7% Miss Rate: Across 1,415 real vulnerabilities, the scanner missed only 38 test cases, with false negatives concentrated narrowly in Weak Hash (25) and Command Injection (12).
Controlled False Positive Rate (13.36%): On safe negative controls, Cyforta 1.0 maintained 86.6% Specificity, correctly recognizing 1,148 out of 1,325 benign test cases without false alarms.
High Overall Discriminative Power (Youden Index = 0.840): Significantly outperforms traditional SAST tools (typical Youden Index: 0.26 – 0.33) and general foundation models (0.10 – 0.25).

Discussion & Enterprise Impact #

Addressing Developer Alert Fatigue

The combination of 97.3% sensitivity and 88.61% precision directly addresses the core operational failure of legacy SAST: noise volume. When security alerts are accompanied by sandboxed exploit verification, security teams transition from triaging false positives to verifying automated remediation.

Interpreting Custom Sanitization Logic

Traditional tools rely on hardcoded registries of known sanitization routines. Modern enterprise software regularly implements domain-specific encoding and validation. Agentic reasoning enables Cyforta 1.0 to interpret custom defenses in context, preventing safe logic from triggering false alarms.

Deployment Economics and Scaling

While multi-stage agentic reasoning incurs higher per-scan compute costs than simple static linters, the downstream elimination of manual triage delivers a compelling net operational ROI in enterprise environments.

Conclusion #

The benchmark results on the OWASP Benchmark v1.2 establish that agentic LLM architectures with automated exploit verification surpass the diagnostic ceiling of traditional static analysis. Achieving a Youden’s Index of 0.840 and 97.3% Sensitivity, Cyforta 1.0 demonstrates that autonomous, machine-speed AppSec can be deployed with high precision in modern CI/CD pipelines.

Citation BibTeX

@techreport{cyforta2026owasp,
  title       = {Cyforta Agentic Scanning on the OWASP Benchmark},
  author      = {{Cyforta Autonomous Security Research Team}},
  institution = {Cyforta Inc.},
  year        = {2026},
  month       = {August},
  version     = {1.0},
  url         = {https://cyforta.com/whitepaper.html},
  note        = {Evaluated on OWASP Benchmark v1.2 (2,740 test cases, Youden Index: 0.840)}
}

Cyforta Agentic Scanning on the OWASP Benchmark