| --- |
| tags: |
| - ml-intern |
| --- |
| # VulnIA V2 — Java Security Dataset Scraper |
|
|
| Pipeline complet de scraping, extraction, nettoyage et préparation d'un dataset Java pour continual fine-tuning de CodeBERT V2. |
|
|
| ## Objectif |
|
|
| - Labels V1 (0-65) conservés inchangés |
| - Nouveaux labels : CRYPTO=66, SAFE=67 |
| - 4 familles cibles : XXE, Deserialization, Crypto, SAFE |
| - Minimum 1500 exemples par famille |
| - Déduplication SHA256, scoring confiance ≥ 0.75, pas de leakage |
|
|
| ## Structure |
|
|
| ``` |
| vulnia-v2/ |
| ├── config/ |
| │ └── scraping_queries.yaml # Requêtes GitHub par famille/sous-type |
| ├── scraper/ |
| │ ├── __init__.py |
| │ ├── constants.py # Patterns API, CWE mapping |
| │ ├── utils.py # Nettoyage code Java, hash, leakage |
| │ ├── ghsa_client.py # Client API GHSA + synthetic fallback |
| │ ├── reference_extractor.py # Extraction URLs GitHub |
| │ ├── code_fetcher.py # Fetch fichiers .java |
| │ ├── fence_extractor.py # Extraction code fences |
| │ ├── reporter.py # Rapport qualité |
| │ └── stages/ # Scripts de pipeline |
| ├── run_pipeline.py # Orchestrateur unifié |
| ├── scrape_vulnia_v2.py # Pipeline principal (synthetic + nettoyage) |
| ├── verify_vulnia_v2.py # Vérification dataset final |
| ├── vulnia_v2_label_mapping.json # Mapping labels 0-67 |
| ├── requirements.txt |
| └── README.md |
| ``` |
|
|
| ## Installation |
|
|
| ```bash |
| git clone https://huggingface.co/MaryamEl/vulnia-v2-scraper |
| cd vulnia-v2-scraper |
| pip install -r requirements.txt |
| ``` |
|
|
| ## Usage |
|
|
| ### 1. Pipeline complet (avec token GitHub) |
|
|
| ```bash |
| export GITHUB_TOKEN="ghp_xxxxxxxx" |
| python run_pipeline.py --stage all |
| ``` |
|
|
| ### 2. Stages individuels |
|
|
| ```bash |
| python run_pipeline.py --stage collect-advisories |
| python run_pipeline.py --stage dedup-advisories |
| python run_pipeline.py --stage extract-advisory-references |
| python run_pipeline.py --stage fetch-advisory-code |
| python run_pipeline.py --stage extract-advisory-code-fences |
| python run_pipeline.py --stage advisory-report |
| ``` |
|
|
| ### 3. Mode dry-run (sans token) |
|
|
| ```bash |
| python run_pipeline.py --stage all --dry-run |
| ``` |
|
|
| ### 4. Génération dataset synthetic uniquement |
|
|
| ```bash |
| python scrape_vulnia_v2.py --skip-github |
| ``` |
|
|
| ### 5. Vérification dataset final |
|
|
| ```bash |
| python verify_vulnia_v2.py --dataset output/vulnia_v2_balanced.csv |
| ``` |
|
|
| ## Outputs |
|
|
| | Fichier | Description | |
| |---------|-------------| |
| | `output/vulnia_v2_balanced.csv` | Dataset principal training-ready | |
| | `output/train.csv` | 70% stratifié | |
| | `output/val.csv` | 15% stratifié | |
| | `output/test.csv` | 15% stratifié | |
| | `output/test_safe_heavy.csv` | Test spécial SAFE hard negatives | |
| | `data/processed/advisories_deduplicated.csv` | **Seed-only** — NE PAS utiliser pour training | |
| | `reports/advisory_seed_quality_report.md` | Verdict readiness | |
|
|
| ## Label Mapping |
|
|
| - V1 labels : 0-65 (inchangés) |
| - CRYPTO : 66 |
| - SAFE : 67 |
|
|
| Voir `vulnia_v2_label_mapping.json` pour le mapping complet. |
|
|
| ## Auteur |
|
|
| ML Intern — Projet VulnIA |
|
|
| <!-- ml-intern-provenance --> |
| ## Generated by ML Intern |
|
|
| This model repository was generated by [ML Intern](https://github.com/huggingface/ml-intern), an agent for machine learning research and development on the Hugging Face Hub. |
|
|
| - Try ML Intern: https://smolagents-ml-intern.hf.space |
| - Source code: https://github.com/huggingface/ml-intern |
|
|
