| --- |
| library_name: transformers |
| tags: |
| - finance |
| - sentiment-analysis |
| - text-classification |
| - market-impact |
| - gated-fusion |
| - multitask-learning |
| - event-study |
| datasets: |
| - NickyNicky/finance-financialmodelingprep-stock-news-sentiments-rss-feed |
| - siddharthmb/stocks-ohlcv |
| license: mit |
| --- |
| |
| # FinImpact Direction1D V4 |
|
|
| This repository contains a custom Hugging Face/PyTorch model for financial text impact modeling. |
| It combines a financial language encoder with market context features through a gated fusion head. |
| This v4 checkpoint is trained primarily as a 1-day market-impact signal model and adds return-aware auxiliary losses for directional-score alignment and confidence calibration. |
|
|
| ## Task |
|
|
| Input: |
|
|
| - Financial news title/body |
| - Symbol-level market context available before the event date |
|
|
| Outputs: |
|
|
| - `direction_1d`: bearish / neutral / bullish future 1-day abnormal market reaction |
| - auxiliary diagnostics: `sentiment`, `direction_5d`, and `volatility` heads are still exported, but their default loss weights are zero in this v4 run. |
|
|
| ## Architecture |
|
|
| - Encoder: `microsoft/deberta-v3-large` |
| - Pooling: attention-mask-aware mean pooling over `last_hidden_state` |
| - Text branch: LayerNorm -> Linear -> GELU -> Dropout |
| - Numeric branch: LayerNorm -> Linear -> GELU -> Dropout |
| - Fusion: learned sigmoid gate between text and numeric branches |
| - Heads: 1D direction classification head plus auxiliary diagnostic heads |
| - Return-aware training: `P(bullish) - P(bearish)` is softly aligned to realized 1-day abnormal return sign/magnitude |
|
|
| This is intentionally not an embedding-only classifier. The model fine-tunes a contextual encoder and learns a supervised 1-day market-impact boundary. |
|
|
| ## Training Data |
|
|
| News source: |
|
|
| - `NickyNicky/finance-financialmodelingprep-stock-news-sentiments-rss-feed` |
| - Fields used: `symbol`, `publishedDate`, `title`, `text`, `sentiment`, `sentimentScore` |
|
|
| Price source: |
|
|
| - `siddharthmb/stocks-ohlcv` |
| - Daily close data used to build lagged market features and future-return labels. |
|
|
| Training rows after join: `13930` |
|
|
| Time split: |
|
|
| - train: `10865` |
| - validation: `1532` |
| - test: `1533` |
|
|
| Symbols: |
|
|
| `AAPL, ABNB, ADBE, ADI, ADP, AMAT, AMD, AMZN, ARKK, ASML, AVGO, BA, BABA, BAC, BIDU, BMY, CAT, CMCSA, COIN, COST, CRM, CSCO, CVX, DHR, DIS, ELF, GE, GM, GOOGL, HD, IBM, INTC, IOVA, JNJ, JPM, KO, LULU, MA, MCD, MDB, META, MRNA, MSFT, MU, NFLX, NKE, NVDA, ORCL, PEP, PFE, PLTR, PYPL, QCOM, RCL, SHOP, SNOW, SPOT, TGT, TSLA, TSM, TXN, UAL, UBER, UNH, UPS, WMT, XOM, ZM` |
|
|
| ## Numeric Features |
|
|
| - `source_sentiment_score` |
| - `sentiment_abs_score` |
| - `site_benzinga` |
| - `site_globenewswire` |
| - `site_yahoo_finance` |
| - `site_marketwatch` |
| - `text_char_len` |
| - `title_char_len` |
| - `is_after_close` |
| - `is_premarket` |
| - `is_market_hours` |
| - `published_hour_sin` |
| - `published_hour_cos` |
| - `ret_1d_lag` |
| - `ret_5d_lag` |
| - `ret_20d_lag` |
| - `vol_5d` |
| - `vol_20d` |
| - `vol_60d` |
| - `vol_ratio_5_20` |
| - `drawdown_20d` |
| - `volume_z_20d` |
| - `sector_code` |
| - `market_ret_1d_lag` |
| - `market_ret_5d_lag` |
| - `sector_ret_1d_lag` |
| - `sector_ret_5d_lag` |
| - `beta_market_60d` |
| - `beta_sector_60d` |
|
|
| All numeric features are fit/scaled on the training split only. |
|
|
| ## Label Construction |
|
|
| - `sentiment` comes from the source dataset label. |
| - `direction_1d` is generated from future 1-day beta-adjusted abnormal log return after the event anchor date. |
| - `direction_5d` is generated from future 5-day beta-adjusted abnormal log return. |
| - A return is neutral when it falls inside a volatility-adjusted threshold. |
| - `volatility` is based on train-split tertiles of future absolute 5-day return. |
|
|
| Event anchoring shifts after-close UTC news to the next calendar day and uses the next available trading date. |
| Duplicate same-symbol same-day event titles are removed before training. |
|
|
| ## Test Metrics |
|
|
| ```json |
| { |
| "sentiment": { |
| "accuracy": 0.009784735812133072, |
| "macro_precision": 0.0032615786040443573, |
| "macro_recall": 0.3333333333333333, |
| "macro_f1": 0.006459948320413436, |
| "confusion_matrix": [ |
| [ |
| 0, |
| 110, |
| 0 |
| ], |
| [ |
| 0, |
| 15, |
| 0 |
| ], |
| [ |
| 0, |
| 1408, |
| 0 |
| ] |
| ], |
| "predicted_class_counts": { |
| "bearish": 0, |
| "neutral": 1533, |
| "bullish": 0 |
| }, |
| "true_class_counts": { |
| "bearish": 110, |
| "neutral": 15, |
| "bullish": 1408 |
| }, |
| "expected_calibration_error": 0.8247767172447622, |
| "brier_score": 1.5716914553389048 |
| }, |
| "direction_1d": { |
| "accuracy": 0.3333333333333333, |
| "macro_precision": 0.2797514006239241, |
| "macro_recall": 0.30693224852151285, |
| "macro_f1": 0.28541099148120913, |
| "confusion_matrix": [ |
| [ |
| 31, |
| 217, |
| 212 |
| ], |
| [ |
| 54, |
| 138, |
| 244 |
| ], |
| [ |
| 124, |
| 171, |
| 342 |
| ] |
| ], |
| "predicted_class_counts": { |
| "bearish": 209, |
| "neutral": 526, |
| "bullish": 798 |
| }, |
| "true_class_counts": { |
| "bearish": 460, |
| "neutral": 436, |
| "bullish": 637 |
| }, |
| "expected_calibration_error": 0.03496117605077468, |
| "brier_score": 0.6723385830557309 |
| }, |
| "direction_5d": { |
| "accuracy": 0.45857795172863663, |
| "macro_precision": 0.28120269843193446, |
| "macro_recall": 0.33741824434028733, |
| "macro_f1": 0.30491929241175314, |
| "confusion_matrix": [ |
| [ |
| 0, |
| 123, |
| 143 |
| ], |
| [ |
| 0, |
| 553, |
| 311 |
| ], |
| [ |
| 0, |
| 253, |
| 150 |
| ] |
| ], |
| "predicted_class_counts": { |
| "bearish": 0, |
| "neutral": 929, |
| "bullish": 604 |
| }, |
| "true_class_counts": { |
| "bearish": 266, |
| "neutral": 864, |
| "bullish": 403 |
| }, |
| "expected_calibration_error": 0.024249568105288305, |
| "brier_score": 0.633983250156182 |
| }, |
| "volatility": { |
| "accuracy": 0.39921722113502933, |
| "macro_precision": 0.13307240704500978, |
| "macro_recall": 0.3333333333333333, |
| "macro_f1": 0.1902097902097902, |
| "confusion_matrix": [ |
| [ |
| 0, |
| 534, |
| 0 |
| ], |
| [ |
| 0, |
| 612, |
| 0 |
| ], |
| [ |
| 0, |
| 387, |
| 0 |
| ] |
| ], |
| "predicted_class_counts": { |
| "lower_vol": 0, |
| "normal_vol": 1533, |
| "higher_vol": 0 |
| }, |
| "true_class_counts": { |
| "lower_vol": 534, |
| "normal_vol": 612, |
| "higher_vol": 387 |
| }, |
| "expected_calibration_error": 0.1760233100613036, |
| "brier_score": 0.712886643106662 |
| }, |
| "event_backtest": { |
| "split": "test", |
| "transaction_cost_bps": 10.0, |
| "active_1d_fraction": 0.6568819308545336, |
| "hit_ratio_1d_active": 0.5789473684210527, |
| "avg_signal_return_1d_all": 0.0023507131510749455, |
| "avg_signal_return_1d_all_net": 0.0016938312202204116, |
| "avg_signal_return_1d_active": 0.0035785931088360393, |
| "avg_signal_return_1d_active_net": 0.002578593108836038, |
| "avg_abnormal_signal_return_1d_all": 0.0007162017627715418, |
| "avg_abnormal_signal_return_1d_all_net": 5.93198319170076e-05, |
| "avg_abnormal_signal_return_1d_active": 0.0010903051661656142, |
| "avg_abnormal_signal_return_1d_active_net": 9.030516616561321e-05, |
| "soft_signal_backtest_1d": { |
| "mean_abs_signal": 0.049029190093278885, |
| "active_fraction_abs_score_ge_0.10": 0.11545988258317025, |
| "hit_ratio_all_gross": 0.532941943900848, |
| "hit_ratio_active_gross": 0.4463276836158192, |
| "avg_signal_return_all": -4.1903094825102016e-05, |
| "avg_signal_return_all_net": -9.093229164136574e-05, |
| "avg_abnormal_signal_return_all": 6.6492548285168596e-06, |
| "avg_abnormal_signal_return_all_net": -4.237994289724156e-05 |
| }, |
| "active_5d_fraction": 0.39399869536855836, |
| "hit_ratio_5d_active": 0.3708609271523179, |
| "avg_signal_return_5d_all": -0.004762698527096634, |
| "avg_signal_return_5d_active": -0.012088107354369436, |
| "confidence_backtests_1d": { |
| "top_10pct_directional_confidence": { |
| "num_events": 153, |
| "mean_confidence": 0.40472039580345154, |
| "hit_ratio": 0.48366013071895425, |
| "avg_signal_return": -0.0017291086948829782, |
| "avg_signal_return_net": -0.002729108694882979, |
| "median_signal_return": -0.0011537353275343776, |
| "avg_abnormal_signal_return": 0.0023775517351216227, |
| "avg_abnormal_signal_return_net": 0.001377551735121622 |
| }, |
| "top_20pct_directional_confidence": { |
| "num_events": 307, |
| "mean_confidence": 0.39327916502952576, |
| "hit_ratio": 0.46579804560260585, |
| "avg_signal_return": -0.0018346647964980064, |
| "avg_signal_return_net": -0.0028346647964980075, |
| "median_signal_return": -0.001494303229264915, |
| "avg_abnormal_signal_return": 0.00013626769358741198, |
| "avg_abnormal_signal_return_net": -0.0008637323064125889 |
| }, |
| "top_30pct_directional_confidence": { |
| "num_events": 460, |
| "mean_confidence": 0.3858344554901123, |
| "hit_ratio": 0.5021739130434782, |
| "avg_signal_return": 0.000168503345380684, |
| "avg_signal_return_net": -0.0008314966546193169, |
| "median_signal_return": 0.00047751690726727247, |
| "avg_abnormal_signal_return": -0.0006967808645482443, |
| "avg_abnormal_signal_return_net": -0.0016967808645482452 |
| }, |
| "threshold_0.34": { |
| "num_events": 1108, |
| "coverage": 0.7227658186562296, |
| "hit_ratio": 0.5523465703971119, |
| "avg_signal_return": 0.002400788538323842, |
| "avg_signal_return_net": 0.001400788538323841, |
| "median_signal_return": 0.0029742957558482885, |
| "avg_abnormal_signal_return": 0.00043566977963306825, |
| "avg_abnormal_signal_return_net": -0.0005643302203669326 |
| }, |
| "threshold_0.36": { |
| "num_events": 636, |
| "coverage": 0.41487279843444225, |
| "hit_ratio": 0.5471698113207547, |
| "avg_signal_return": 0.001725182487273244, |
| "avg_signal_return_net": 0.000725182487273243, |
| "median_signal_return": 0.0027859483379870653, |
| "avg_abnormal_signal_return": -0.0001395135721791607, |
| "avg_abnormal_signal_return_net": -0.0011395135721791617 |
| }, |
| "threshold_0.38": { |
| "num_events": 247, |
| "coverage": 0.16112198303979125, |
| "hit_ratio": 0.44129554655870445, |
| "avg_signal_return": -0.0028902063559107334, |
| "avg_signal_return_net": -0.0038902063559107343, |
| "median_signal_return": -0.0022296553943306208, |
| "avg_abnormal_signal_return": 0.0008437850360328761, |
| "avg_abnormal_signal_return_net": -0.00015621496396712474 |
| }, |
| "threshold_0.40": { |
| "num_events": 85, |
| "coverage": 0.055446836268754074, |
| "hit_ratio": 0.43529411764705883, |
| "avg_signal_return": -0.0034284473146887168, |
| "avg_signal_return_net": -0.004428447314688717, |
| "median_signal_return": -0.0031941309571266174, |
| "avg_abnormal_signal_return": 0.0008126156158087884, |
| "avg_abnormal_signal_return_net": -0.00018738438419121242 |
| }, |
| "threshold_0.42": { |
| "num_events": 20, |
| "coverage": 0.01304631441617743, |
| "hit_ratio": 0.25, |
| "avg_signal_return": -0.008969856356270612, |
| "avg_signal_return_net": -0.009969856356270613, |
| "median_signal_return": -0.011959618888795376, |
| "avg_abnormal_signal_return": -0.0006033775032847188, |
| "avg_abnormal_signal_return_net": -0.0016033775032847197 |
| }, |
| "threshold_0.45": { |
| "num_events": 2, |
| "coverage": 0.001304631441617743, |
| "hit_ratio": 0.5, |
| "avg_signal_return": 0.0010717622935771942, |
| "avg_signal_return_net": 7.176229357719333e-05, |
| "median_signal_return": 0.0010717622935771942, |
| "avg_abnormal_signal_return": 0.01302456425037235, |
| "avg_abnormal_signal_return_net": 0.01202456425037235 |
| }, |
| "threshold_0.50": { |
| "num_events": 0, |
| "coverage": 0.0, |
| "hit_ratio": null, |
| "avg_signal_return": null, |
| "avg_signal_return_net": null, |
| "median_signal_return": null, |
| "avg_abnormal_signal_return": null, |
| "avg_abnormal_signal_return_net": null |
| } |
| }, |
| "mean_text_gate_weight": 0.506599485874176, |
| "std_text_gate_weight": 0.0016385371563956141 |
| } |
| } |
| ``` |
|
|
| The `confidence_backtests_1d` section ranks events by directional confidence, where directional confidence is `max(P(bearish), P(bullish))`. |
| This is the preferred way to inspect whether the model is useful as a signal filter. |
| V4 also reports cost-adjusted backtest fields using `10.0` bps per active directional event. |
|
|
| ## Important Limitations |
|
|
| This is a research model, not trading advice. |
|
|
| Known limitations: |
|
|
| - Public news sentiment labels can be noisy. |
| - Daily OHLCV alignment is an approximation; intraday timestamp alignment would be better. |
| - Market-direction labels are derived from future returns and are sensitive to threshold choice. |
| - Results should be evaluated out-of-sample by date and by ticker before any practical use. |
|
|
| ## Files |
|
|
| - `pytorch_model.bin`: custom gated-fusion model weights. |
| - `training_config.json`: training and dataset configuration. |
| - `feature_schema.json`: numeric feature scaler and schema. |
| - `label_mapping.json`: task label names. |
| - `metrics.json`: train/validation/test metrics. |
| - `test_predictions.csv`: event-level test predictions and returns. |
| - `top_10pct_directional_confidence_1d` in `test_predictions.csv`: marks the highest-confidence directional event subset. |
| - `confusion_matrix_*.csv`: confusion matrices per split/task. |
|
|
