ThreatHunter v5.1 — 全域 Pipeline 流程圖

完整資料流 · Defense-in-Depth · OSV-First Intelligence · 2026-04-19
7
AI Agents
20
Skill SOPs
10
Tools
6+1
Rust Crates
4
Scan Paths
6/6
API-Driven
4
CVE Filters
👤
使用者輸入
Browser → HTTP
🖥 主介面
ui/static/index.html
HTML5CSS3JS
⚡ SSE 即時串流
ui/static/app.js
Vanilla JSEventSource
🔬 Checkpoint 儀表板
ui/static/checkpoint.html
HTMLJS
POST /api/scan { tech_stack, input_type }
🚀
FastAPI Server + UI Filter
ui/server.py
🚀 FastAPI + Uvicorn
ui/server.py — _pipeline_worker()
Python 3.12 FastAPI SSE threading
接收掃描請求 → 分配 scan_id → 開啟 Worker Thread → SSE 推送進度
🛡 UI CVE Filter
_enrich_result() L165
DEFENSE LAYER 4 · CVE < 2005 → 隱藏
最後防線:從 vulnerability_detail 濾除遠古 CVE
Worker Thread → run_pipeline_with_callback()
🛡
L0 安全閘門 — 確定性(LLM 之前)
OWASP LLM01:2025
🔍 Input Sanitizer
input_sanitizer.py
Python regex hashlib
Blocklist 掃描 → L0 正則標記 → 輸入類型推斷 → SHA-256 追蹤
🦀 Rust L0 Core
rust/sanitizer/
計劃中PyO3
SanitizeResult { safe, input_type, hash }
⚙️
Pipeline v3.1 Coordinator
main.py
🧭 Orchestrator + Path Router
agents/orchestrator.py + main.py
Python CrewAI
📦 Path A: PKG 🔍 Path B: CODE 🤖 Path B: INJECTION ⚙️ Path C: CONFIG
input_type → 動態選擇 Skill SOP → Path-Aware 路由
📝 Checkpoint Recorder
checkpoint.py
PythonJSONL
STAGE_ENTER · LLM_CALL · TOOL_CALL · DEGRADATION
append-only · threading.Lock
OSV Batch Warmup (Harness 0) → PackageExtractor
Layer 1 — 並行處理(同時執行)
concurrent.futures
🔒 Security Guard
agents/security_guard.py
CrewAI ast.parse() regex
Dual LLM Pattern · allow_delegation=False
→ skills/security_guard.md
🧮 Intel Fusion
agents/intel_fusion.py
CrewAI OSV-First 6D Score
DEFENSE LAYER 2 · Harness 2.5 CVE < 2005 → 過濾
search_osv 優先 · NVD fallback · calculate_composite_score()
→ skills/intel_fusion.md
parallel output → merged into pipeline context
🔭
Scout Agent — 情報蒐集
agents/scout.py
🔭 Scout Agent
agents/scout.py — run_scout_pipeline()
CrewAI OSV-First SKILL_MAP
DEFENSE LAYER 1 · Harness 3.5 CVE < 2005 → 過濾
Harness 0: OSV Batch Cache Warmup · Harness 3: NVD 驗證 (OSV cache 優先)
Harness 3.5: CVE year filter · 從 output 移除遠古漏洞
📋 Path-Aware SOP
skills/
pkg → threat_intel.md code → source_code_audit.md injection → ai_security_audit.md config → config_audit.md
Scout JSON { vulnerabilities[], packages[], risk_hints }
🔬
Analyst Agent — 攻擊連鎖分析
agents/analyst.py
🔬 Analyst Agent
agents/analyst.py — run_analyst_pipeline()
CrewAI 3-Task Pipeline SKILL_MAP
Harness: _harness_filter_ancient_cves() · 標記 NEEDS_VERIFICATION
Task 1: Parse Scout → Task 2: KEV + Exploit 查詢 → Task 3: Chain Analysis
KillChain · 攻擊連鎖推理 · Adaptive Weight
📋 Path-Aware SOP
skills/
pkg → chain_analysis.md code → code_chain_analysis.md injection → ai_chain_analysis.md config → config_chain_analysis.md
Analyst JSON { analysis[], chain_risk, adjusted_risk }
⚔️
Critic Agent — 對抗式辯論 (ColMAD)
agents/critic.py · ENABLE_CRITIC=True
⚔️ Critic Agent
agents/critic.py — run_critic_pipeline()
CrewAI ColMAD Debate 5D Scorecard
red_team · blue_team · arbiter 三角色驗證
MAINTAIN / DOWNGRADE 判決 · challenge 產生 critic_context
📋 Path-Aware SOP
skills/
pkg → debate_sop.md code → code_debate_sop.md injection → ai_debate_sop.md config → config_debate_sop.md
Critic JSON { verdict, challenges[], weighted_score }
📋
Advisor Agent — 最終裁決者 (Judge)
agents/advisor.py
📋 Advisor Agent
agents/advisor.py — run_advisor_pipeline()
CrewAI 6-Layer Harness Judge + Feedback
DEFENSE LAYER 3 · Harness 6 CVE < 2005 → 從 urgent/important 移除
L1: 強制輸出 · L2: Schema · L3: risk_score 0-100 · L4: command · L5: 重複比對 · L6: 年份過濾
📋 Path-Aware SOP
skills/
pkg → action_report.md code → code_action_report.md injection → ai_action_report.md config → config_action_report.md
Advisor JSON → _enrich_result() → SSE scan_complete → UI
🛡 Defense-in-Depth — CVE 年份過濾(year < 2005 → 攔截)
L1
Scout
scout.py Harness 3.5
Scout output 過濾
L2
Intel Fusion
intel_fusion.py Harness 2.5
fusion_results 過濾
L3
Advisor
advisor.py Harness 6
urgent/important 過濾
L4
UI Server
server.py _enrich_result
vulnerability_detail 最終過濾
⚡ 六維情報融合分析 — 100% API 驅動 · 0% LLM 猜測
CVSS
20%
NVD API v2
EPSS
30%
FIRST.org
KEV
25%
CISA API
GHSA
10%
OSV.dev
ATT&CK
10%
CWE→CAPEC Map
OTX
5%
AlienVault
🔧
工具層 — Agents 呼叫
tools/
PRIMARY
🎯 OSV Tool
osv_tool.py
ecosystem-aware
FALLBACK
🔎 NVD Tool
nvd_tool.py
⚠️ KEV Tool
kev_tool.py
🌐 OTX Tool
otx_tool.py
💣 Exploit Tool
exploit_tool.py
📊 EPSS Tool
epss_tool.py
🎯 ATT&CK Tool
attck_tool.py
📦 GHSA Tool
ghsa_tool.py
💾 Memory Tool
memory_tool.py
📦 Pkg Extractor
package_extractor.py
🗃
持久化記憶 + 快取
memory/ · data/ · logs/
📂 Scout
scout_memory.json
📂 Analyst
analyst_memory.json
📂 Advisor
advisor_memory.json
📝 Checkpoint
*.jsonl
append-only
🎯 OSV Cache
osv_cache_*.json
24h TTL
📦 NVD Cache
nvd_cache.json
📊 EPSS Cache
epss_cache_*.json
🏗
Harness Engineering 基礎設施
三柱架構
Layer 1 — Context
harness/context/
agent_map · doc_indexer
Layer 2 — Constraints
harness/constraints/
arch_linter · boundary_rules.toml
Layer 3 — Entropy
harness/entropy/
entropy_scanner · until_clean_loop
🔧 Config / LLM
config.py
OpenRouter
降級瀑布 · 429 輪替 · rate_limiter
🦀
Rust 高效能層 + 安全隔離 Sandbox
rust/ · sandbox/ · PyO3 + maturin
🦀 Rust PyO3 Crates(6 個 cdylib · Cargo Workspace)
🛡 sanitizer
rust/sanitizer/
Rust PyO3 regex sha2
scan_blocklist · infer_input_type · sha256_hex
替代 input_sanitizer.py 的效能關鍵路徑
📝 checkpoint_writer
rust/checkpoint_writer/
Rust tokio parking_lot
open_writer · write_line · flush · write_batch
Mutex + BufWriter 高效 JSONL I/O
🔒 prompt_sandbox
rust/prompt_sandbox/
Rust wasmtime 28 WASM
sandbox_eval · L0.5 WASM Runtime
wasmtime Host · prompt_guard.wasm
📋 json_validator
rust/json_validator/
serde_json regex
safe_parse_json · validate_cve_id · validate_cve_list
💾 memory_validator
rust/memory_validator/
serde_json regex
validate_memory_write · validate_cve_id
🔗 url_builder
rust/url_builder/
url crate
build_api_url · validate_url · encode_query_value
🎯 prompt_sandbox_guest
rust/prompt_sandbox_guest/
wasm32-unknown-unknown
WASM Guest → prompt_guard.wasm
🐳 Python Sandbox 安全隔離模組(三層防禦 + Graceful Degradation)
🔍 AST Guard
sandbox/ast_guard.py
Python ast.parse()
Layer 1 — AST 遮罩 + timeout
🐳 Docker Sandbox
sandbox/docker_sandbox.py
Python Docker
Layer 2 — 容器隔離 · seccomp-profile.json
🧹 Memory Sanitizer
sandbox/memory_sanitizer.py
Python regex
Layer 3 — 記憶快取淨化 · CVE year 正則
BUILD maturin develop MinGW-w64 GNU x86_64-pc-windows-gnu build_rust_crates.py
☁️
外部 API 依賴
REST / JSON Feed
🎯 OSV.dev
api.osv.dev/v1
PRIMARY
🔒 NVD v2
services.nvd.nist.gov
⚠️ CISA KEV
cisa.gov
📊 FIRST EPSS
api.first.org/epss
🌐 OTX
otx.alienvault.com
🎯 ATT&CK
CWE→CAPEC→T-ID
🤖 LLM
openrouter.ai
💣 GitHub
api.github.com