Model Evaluation
Complete performance analysis — Random Forest trained on 80,791 financial tweets with FinBERT auto-labelling
Test Accuracy
--
held-out test set
CV Accuracy
--
5-fold mean
AUC-ROC
--
binary class
F1 Score
--
weighted avg
Precision
--
weighted avg
Recall
--
weighted avg
Confusion matrix
-- predictions
← Predicted label →
Actual label ↕
Predicted Rise
Predicted Fall
--% of Rise
--
True Positive
Actual Rise
Predicted Rise ✓
Predicted Rise ✓
--% of Rise
--
False Negative
Actual Rise
Predicted Fall ✗
Predicted Fall ✗
--% of Fall
--
False Positive
Actual Fall
Predicted Rise ✗
Predicted Rise ✗
--% of Fall
--
True Negative
Actual Fall
Predicted Fall ✓
Predicted Fall ✓
Sensitivity
--
TP/(TP+FN)
Specificity
--
TN/(TN+FP)
MCC
--
Matthews Corr.
ROC curve
AUC = --
Interpretation
AUC > 0.7 indicates good discriminative ability. Our model -- the 0.7 threshold, suggesting the Random Forest meaningfully differentiates Rise vs Fall signals beyond random chance.
Classification report
| Class | Precision | Recall | F1-Score | Support |
|---|---|---|---|---|
| Rise (1) | -- | -- | -- | -- |
| Fall (0) | -- | -- | -- | -- |
| Weighted avg | -- | -- | -- | -- |
5-fold cross-validation
Mean: --
Fold 1
--
Fold 2
--
Fold 3
--
Fold 4
--
Fold 5
--
Mean: --
Std: --
Stratified K-Fold
Feature importance (Gini impurity)
Model configuration
AlgorithmRandom Forest
n_estimators200
max_depth12
max_featuressqrt
min_samples_split5
class_weightbalanced
criteriongini
SMOTEenabled
train/test split80% / 20%
Label sourceFinBERT auto
NLP modelProsusAI/FinBERT
Total features36