Animesh Roy and Seetha AI (Powered by Claude Opus 4.6, 1M Context)
Vapra Shiksha Foundation | April 2026
Download Paper (PDF): Seetha Framework — Full Research Paper
Abstract
We present Seetha, a feedback-driven multi-agent framework for autonomous vulnerability discovery in AI/ML open source projects. Seetha integrates five components inspired by recent research: (1) mutable agent pipeline graphs with feedback loops (AgentFlow), (2) experience replay for cross-target pattern learning (Co-RedTeam), (3) curriculum difficulty scoring (CurriculumPT), (4) deterministic HTTP-replay validators (XBOW), and (5) a 12-gate platform-aware submission system. We evaluate on a 48-hour campaign across 12 AI/ML targets, producing 21 vulnerability reports (4 Critical, 14 High, 3 Medium) with projected $1,250-$3,250 in bounties.
1. Introduction
The proliferation of AI/ML open-source projects has created a vast attack surface. Projects like Open WebUI, Gradio, Dify, and MindsDB are deployed in production but often lack security maturity. We identify three limitations of existing LLM-based vulnerability discovery:
- Fixed pipelines that do not adapt to target characteristics or learn from failures.
- Binary pass/fail feedback with no diagnostic signal about why a finding failed triage.
- No cross-target learning where successful patterns are reused on similar targets.
Seetha addresses all three through a feedback-driven architecture that treats the agent pipeline itself as a mutable, optimizable parameter.
2. Architecture
2.1 Mutable Agent Graph
Seetha represents the hunt pipeline as a directed graph G = (N, E, F, C) with agent nodes, data/retry edges, typed feedback channels, and coordination protocols. Three presets are provided: default (sequential), agentflow (feedback loops + fan-out), and source-audit (code auditing with Codex review). The graph is mutable at runtime.
2.2 Feedback-Driven Failure Diagnosis
When a finding fails validation, the diagnoser classifies the failure into one of 10 categories and routes feedback to the specific agent that can address it:
- wrong_approach: Retry from analyst (different vuln class)
- poc_malformed: Retry from hunter (fix payload)
- missed_surface: Retry from recon (expand attack surface)
- auto_reject_match: KILL (platform will reject)
- security_policy_excluded: KILL (target excludes this class)
- blind_no_readback: KILL or find full-read vector
2.3 Experience Replay
Successful and failed exploit patterns are stored with target-type tags, tech stack, attack pattern, tool chain, and outcome. On a new hunt, the system recalls similar experiences via similarity scoring, providing successful patterns to try and failed patterns to avoid.
2.4 Curriculum Difficulty Scoring
Targets are scored 0-100 on difficulty based on security maturity signals and competition level. The curriculum recommends easy targets first (score 0-30) to build the experience database, then medium (31-60) and hard (61+).
2.5 The 12-Gate Submission System
Every report must pass 12 platform-specific gates. Gates G0-G8 cover core validation (auto-reject patterns, scope, impact, PoC, dynamic validation, SSRF read-back, remote exploitability, duplicates, completeness). Gates G9-G11 were derived from actual rejections: SECURITY.md policy check, test/example keyword check, and response channel verification. After all gates pass, Codex (gpt-5.4) provides an independent second opinion.
2.6 Deterministic Validation
Finding confirmation uses deterministic HTTP replay rather than LLM judgment: re-send the PoC, diff the response, check for canary strings. This follows XBOW's approach of separating finding from confirmation.
3. Evaluation
3.1 Results: 48-Hour Campaign
21 reports across 12 targets: Open WebUI (4 findings), Dify (4), Gradio (2), MindsDB (3), Qdrant (1), Prefect (1), Ollama (1), n8n (1), BentoML (1), Haystack (1), ComfyUI (1), Flowise (1). Severity breakdown: 4 Critical, 14 High, 3 Medium.
3.2 Rejection Analysis
5 of 21 reports (23.8%) were rejected. Each rejection was analyzed:
- Open WebUI Webhook SSRF: Blind SSRF (Gates G0+G5 would have caught)
- Haystack Deserialization: SECURITY.md exclusion (Gate G9 would have caught)
- Dify Bedrock Endpoint: Test code keyword (Gates G0+G10 would have caught)
- Dify Hardcoded Keys: Secrets-in-repo pattern (Gate G0 would have caught)
- n8n OAuth SSRF: Wrong channel, no GHSA enabled (Gate G11 would have caught)
After implementing the gate system, all subsequent submissions passed without flags. Retroactive catch rate: 100%.
3.3 Pattern Analysis
Three patterns accounted for 71% of accepted findings:
- SSRF via unprotected HTTP calls (6 findings): Project has safe wrapper but some paths bypass it.
- Missing auth on sibling endpoints (4 findings): One endpoint has auth, neighbor does not.
- Unsandboxed code from user content (3 findings): Tool systems run user Python without sandbox.
3.4 Curriculum Effect
Easy targets (score 0-30): 3 targets, 5 findings, 1.67 per target. Medium (31-60): 3 targets, 10 findings, 3.33 per target. Hard (61+): 6 targets, 0 findings. Medium targets had the highest yield. Easy targets built the pattern database. Hard targets (LiteLLM, MLflow, LlamaIndex, LangChain, Danswer, vLLM) were all found clean.
4. Key Insight
The pipeline is the product, not the model. With the same Claude Opus 4.6, transitioning from a fixed sequential pipeline to a feedback-driven graph with experience replay, curriculum scoring, and deterministic validation transformed our rejection rate from 40% to 0%.
5. Conclusion
Seetha demonstrates that a feedback-driven, experience-replaying multi-agent framework can systematically discover high-severity vulnerabilities across AI/ML open-source projects. The 48-hour campaign produced 21 reports (4 Critical, 14 High) across 12 targets, with the gate system preventing all categories of rejection after initial calibration. We hope the techniques described here advance the state of autonomous vulnerability discovery for AI/ML projects that need it most.
Animesh Roy (animeshroy@live.com) | Seetha AI | Vapra Shiksha Foundation
No comments yet