validmind.ROCCurve

ROCCurve

@tags('sklearn', 'binary_classification', 'multiclass_classification', 'model_performance', 'visualization')

@tasks('classification', 'text_classification')

defROCCurve(model:validmind.vm_models.VMModel,dataset:validmind.vm_models.VMDataset) → Tuple[go.Figure, validmind.vm_models.RawData]:

Evaluates classification model performance by generating and plotting the Receiver Operating Characteristic (ROC) curve and calculating the Area Under Curve (AUC) score, for both binary and multiclass models.

Purpose

The Receiver Operating Characteristic (ROC) curve evaluates the performance of classification models. This curve illustrates the balance between the True Positive Rate (TPR) and False Positive Rate (FPR) across various threshold levels. In combination with the Area Under the Curve (AUC), the ROC curve measures the model's discrimination ability between classes. For binary problems (e.g., default vs non-default) a single curve is drawn for the positive class. For multiclass problems the curve is computed one-vs-rest — one curve and AUC per class, plus a micro-average across all classes — so the model's discrimination ability can be assessed for every class. Ideally, a higher AUC score signifies superior model performance in accurately distinguishing between classes.

Test Mechanism

This test selects the target model and dataset and determines the number of classes from the true labels. For binary targets it computes the predicted probabilities for the positive class and, together with the true outcomes, generates and plots a single ROC curve. For multiclass targets it obtains the full per-class probability matrix from the model and plots a one-vs-rest curve for each class along with a micro-average curve. In both cases a line signifying randomness (AUC of 0.5) is included, and the AUC score(s) are computed as a numerical estimation of performance. If any Infinite values are detected in the ROC threshold, these are effectively eliminated. The resulting ROC curves, AUC scores, and thresholds are consequently saved for future reference.

Signs of High Risk

  • A high risk is potentially linked to the model's performance if the AUC score drops below or nears 0.5.
  • Another warning sign would be the ROC curve lying closer to the line of randomness, indicating no discriminative ability.
  • For the model to be deemed competent at its classification tasks, it is crucial that the AUC score is significantly above 0.5.

Strengths

  • The ROC Curve offers an inclusive visual depiction of a model's discriminative power throughout all conceivable classification thresholds, unlike other metrics that solely disclose model performance at one fixed threshold.
  • Despite the proportions of the dataset, the AUC Score, which represents the entire ROC curve as a single data point, continues to be consistent, proving to be the ideal choice for such situations.

Limitations

  • For multiclass models the curve is computed one-vs-rest (one curve per class plus a micro-average), which requires per-class probabilities from the model's predict_proba. Models that cannot produce a full per-class probability matrix (e.g. metadata-only models, or predictions supplied as a single precomputed probability column) are skipped for the multiclass case.
  • Furthermore, its performance might be subpar with models that output probabilities highly skewed towards 0 or 1.
  • At the extreme, the ROC curve could reflect high performance even when the majority of classifications are incorrect, provided that the model's ranking format is retained. This phenomenon is commonly termed the "Class Imbalance Problem".