Final Paper Suite Summary
Rows: 20
Dataset and Audit
| Benchmark | Matched Pairs | Positives | Negatives | Fraud Users | Benign Users | Templates | Positive Rate | Benign Motif Hit | static_agg_auc | Txn AUC | Idx AUC | Prefix AUC | Time AUC | Account AUC | Active AUC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| easy | 2222.2000 ± 128.4434 | 2222.2000 ± 128.4434 | 2222.2000 ± 128.4434 | 304.6000 ± 23.2551 | 304.6000 ± 23.2551 | 300.2000 ± 24.2425 | 0.5000 ± 0.0000 | 0.0000 ± 0.0000 | 0.5000 ± 0.0000 | 0.5000 ± 0.0000 | 0.5000 ± 0.0000 | 0.5000 ± 0.0000 | 0.5000 ± 0.0000 | 0.5000 ± 0.0000 | 0.5000 ± 0.0000 |
| hard | 2317.6000 ± 21.9727 | 2317.6000 ± 21.9727 | 2317.6000 ± 21.9727 | 315.0000 ± 0.0000 | 315.0000 ± 0.0000 | 315.0000 ± 0.0000 | 0.5000 ± 0.0000 | 0.0000 ± 0.0000 | 0.5000 ± 0.0000 | 0.5000 ± 0.0000 | 0.5000 ± 0.0000 | 0.5000 ± 0.0000 | 0.5000 ± 0.0000 | 0.5000 ± 0.0000 | 0.5000 ± 0.0000 |
| medium | 2356.6000 ± 18.0361 | 2356.6000 ± 18.0361 | 2356.6000 ± 18.0361 | 263.0000 ± 0.0000 | 263.0000 ± 0.0000 | 263.0000 ± 0.0000 | 0.5000 ± 0.0000 | 0.0000 ± 0.0000 | 0.5000 ± 0.0000 | 0.5000 ± 0.0000 | 0.5000 ± 0.0000 | 0.5000 ± 0.0000 | 0.5000 ± 0.0000 | 0.5000 ± 0.0000 | 0.5000 ± 0.0000 |
| oracle_calib | 2606.6000 ± 454.3449 | 2606.6000 ± 454.3449 | 2606.6000 ± 454.3449 | 409.8000 ± 86.5402 | 409.8000 ± 86.5402 | 409.8000 ± 86.5402 | 0.5000 ± 0.0000 | 0.0000 ± 0.0000 | 0.5000 ± 0.0000 | 0.5000 ± 0.0000 | 0.5000 ± 0.0000 | 0.5000 ± 0.0000 | 0.5000 ± 0.0000 | 0.5000 ± 0.0000 | 0.5000 ± 0.0000 |
Probes and Models
| Benchmark | Primary Probe | Secondary Probe | XGB ROC/PR | StaticGNN ROC/PR | SeqGRU Clean ROC/PR | SeqGRU Shuf ROC/PR | SeqGRU Delta |
|---|---|---|---|---|---|---|---|
| easy | MotifProbe 1.0000 ± 0.0000 | RawMotifProbe 0.9983 ± 0.0011 | 0.5000 ± 0.0000 / 0.5000 ± 0.0000 | 0.4946 ± 0.0128 / 0.5001 ± 0.0099 | 1.0000 ± 0.0000 / 1.0000 ± 0.0000 | 0.4997 ± 0.0096 / 0.5008 ± 0.0077 | -0.5003 ± 0.0096 |
| hard | MotifProbe 0.5790 ± 0.0045 | RawMotifProbe 0.5910 ± 0.0105 | 0.5000 ± 0.0000 / 0.5000 ± 0.0000 | 0.5026 ± 0.0198 / 0.5081 ± 0.0128 | 0.6876 ± 0.0128 / 0.7058 ± 0.0250 | 0.4994 ± 0.0033 / 0.4981 ± 0.0031 | -0.1883 ± 0.0111 |
| medium | MotifProbe 0.6374 ± 0.0069 | RawMotifProbe 0.6482 ± 0.0058 | 0.5000 ± 0.0000 / 0.5000 ± 0.0000 | 0.4922 ± 0.0203 / 0.4956 ± 0.0231 | 0.8391 ± 0.0174 / 0.8520 ± 0.0110 | 0.5053 ± 0.0071 / 0.5054 ± 0.0078 | -0.3337 ± 0.0191 |
| oracle_calib | AuditOracle 1.0000 ± 0.0000 | RawMotifOracle 1.0000 ± 0.0000 | 0.5000 ± 0.0000 / 0.5000 ± 0.0000 | 0.5222 ± 0.0235 / 0.5141 ± 0.0202 | 1.0000 ± 0.0000 / 1.0000 ± 0.0000 | 0.4968 ± 0.0043 / 0.5008 ± 0.0057 | -0.5032 ± 0.0043 |
Temporal GNNs
| Benchmark | TGN ROC/PR/Delta | TGAT ROC/PR/Delta | DyRep ROC/PR/Delta | JODIE ROC/PR/Delta |
|---|---|---|---|---|
| easy | 0.5660 ± 0.0123 / 0.5566 ± 0.0107 / -0.0668 ± 0.0126 | 0.5200 ± 0.0119 / 0.5177 ± 0.0086 / -0.0294 ± 0.0244 | 0.5320 ± 0.0090 / 0.5245 ± 0.0100 / -0.0306 ± 0.0150 | 0.5282 ± 0.0085 / 0.5210 ± 0.0079 / -0.0420 ± 0.0231 |
| hard | 0.5060 ± 0.0081 / 0.5069 ± 0.0062 / -0.0006 ± 0.0131 | 0.5157 ± 0.0135 / 0.5133 ± 0.0163 / -0.0145 ± 0.0173 | 0.5229 ± 0.0050 / 0.5172 ± 0.0072 / -0.0171 ± 0.0147 | 0.5137 ± 0.0085 / 0.5087 ± 0.0076 / -0.0231 ± 0.0139 |
| medium | 0.5177 ± 0.0063 / 0.5133 ± 0.0034 / -0.0124 ± 0.0094 | 0.5199 ± 0.0146 / 0.5121 ± 0.0105 / -0.0334 ± 0.0211 | 0.5285 ± 0.0246 / 0.5252 ± 0.0144 / -0.0290 ± 0.0181 | 0.5298 ± 0.0057 / 0.5235 ± 0.0103 / -0.0299 ± 0.0061 |
| oracle_calib | 0.6270 ± 0.0103 / 0.6107 ± 0.0144 / -0.1288 ± 0.0173 | 0.6327 ± 0.0465 / 0.6104 ± 0.0381 / -0.1403 ± 0.0654 | 0.6135 ± 0.0481 / 0.6010 ± 0.0400 / -0.1203 ± 0.0600 | 0.5903 ± 0.0170 / 0.5720 ± 0.0146 / -0.0947 ± 0.0227 |
Runtime
| Benchmark | Run Time (sec) | StaticGNN Eval Time (sec) |
|---|---|---|
| easy | 1345.9431 ± 135.9206 | 198.4129 ± 45.3445 |
| hard | 2613.7203 ± 231.4288 | 272.9988 ± 70.8384 |
| medium | 2181.9480 ± 228.0777 | 184.2332 ± 57.5306 |
| oracle_calib | 1136.5994 ± 283.1015 | 170.9515 ± 34.9402 |
Failed Gate Checks
| Benchmark | Seed | Gate Pass | Volume Failures | Hard Gate Failures | Advisory Failures |
|---|---|---|---|---|---|
| easy | 0 | 1 | - | - | TGN ROC-AUC: 0.5835 (>=0.75) |
| easy | 1 | 1 | - | - | TGN ROC-AUC: 0.5495 (>=0.75) |
| easy | 2 | 1 | - | - | TGN ROC-AUC: 0.5638 (>=0.75) |
| easy | 3 | 1 | - | - | TGN ROC-AUC: 0.5693 (>=0.75) |
| easy | 4 | 1 | - | - | TGN ROC-AUC: 0.5639 (>=0.75) |
| hard | 0 | 0 | - | MotifProbe ROC-AUC: 0.5838 (>=0.99) | MotifProbe pair-sep: 0.1676 (>=0.99) |
| hard | 1 | 0 | - | MotifProbe ROC-AUC: 0.5771 (>=0.99) | MotifProbe pair-sep: 0.1543 (>=0.99) |
| hard | 2 | 0 | - | MotifProbe ROC-AUC: 0.5824 (>=0.99) | MotifProbe pair-sep: 0.1648 (>=0.99) |
| hard | 3 | 0 | - | MotifProbe ROC-AUC: 0.5724 (>=0.99) | MotifProbe pair-sep: 0.1448 (>=0.99) |
| hard | 4 | 0 | - | MotifProbe ROC-AUC: 0.5790 (>=0.99) | MotifProbe pair-sep: 0.1581 (>=0.99) |
| medium | 0 | 0 | - | MotifProbe ROC-AUC: 0.6463 (>=0.99) | MotifProbe pair-sep: 0.2926 (>=0.99) |
| medium | 1 | 0 | - | MotifProbe ROC-AUC: 0.6303 (>=0.99) | MotifProbe pair-sep: 0.2606 (>=0.99) |
| medium | 2 | 0 | - | MotifProbe ROC-AUC: 0.6423 (>=0.99) | MotifProbe pair-sep: 0.2846 (>=0.99) |
| medium | 3 | 0 | - | MotifProbe ROC-AUC: 0.6314 (>=0.99) | MotifProbe pair-sep: 0.2629 (>=0.99) |
| medium | 4 | 0 | - | MotifProbe ROC-AUC: 0.6366 (>=0.99) | MotifProbe pair-sep: 0.2731 (>=0.99) |
| oracle_calib | 0 | 1 | nan | nan | TGN ROC-AUC: 0.6135 (>=0.75) |
| oracle_calib | 1 | 1 | - | - | TGN ROC-AUC: 0.6250 (>=0.75) |
| oracle_calib | 2 | 1 | - | - | TGN ROC-AUC: 0.6353 (>=0.75) |
| oracle_calib | 3 | 1 | - | - | TGN ROC-AUC: 0.6223 (>=0.75) |
| oracle_calib | 4 | 1 | - | - | TGN ROC-AUC: 0.6391 (>=0.75) |