Week 3 Diagnostic Experiments for Disputed Cases: A Deep Dive into Controlled Experiments and Predictive Disagreement

In applied machine learning, especially in high-stakes domains like lending or healthcare, the journey doesn’t end once a model reaches a certain accuracy threshold. Rather, it's crucial to rigorously diagnose the failure modes and ambiguous cases that challenge our systems. In this blog post, I’ll explore Week 3 diagnostic experiments focusing on disputed or edge cases — those records where model and human assessors diverge, or models produce uncertain predictions.

We’ll walk through leveraging tools like disagreement rate and predictive entropy to identify and triage risk-prone samples, discuss how these tie to underlying issues such as distribution shifts, data gaps, and objective mismatches, and share practical approaches like controlled experiments, counterfactual testing, and augmented training to close the loop.

Disagreement Rate as a High-Signal Risk Indicator

First, what do we mean by disagreement rate? In the context of diagnostic experiments, it measures how often two decision makers (for instance, a human labeler and an ML model, or two separate models) produce conflicting outputs on the same samples.

Disagreement is often an overlooked goldmine of information because it directly flags uncertain or high-risk samples:

Ambiguity: Cases near decision boundaries or with genuinely ambiguous features will cause differing interpretations.
Model blind spots: If disagreement is consistently concentrated in certain subgroups, it signals underrepresented data or features missing from model training.
Operational risk: High disagreement cases tend to have higher error rates and may require human review or specialized treatment.

To put this concept into practice, consider a risk-scoring model for loan approvals. By tracking the disagreement rate between the model's predicted risk class and the underwriter’s decision, we can isolate disputes that warrant closer scrutiny, potentially indicating edge cases or distributional shifts.

Measuring Disagreement Rate

Formally, disagreement rate over a dataset D can be defined as:

Notation Description D = x_i, y_i Dataset with inputs and ground truth labels m(x_i) Model prediction h(x_i) Human or second labeler prediction

The disagreement rate is:

Disagreement Rate = (1 / |D|) * Σ_i 1 m(x_i) ≠ h(x_i)

Where 1· is the indicator function that equals 1 if the condition is true, else 0.

Predictive Entropy: Quantifying Model Uncertainty

Complementary to disagreement, predictive entropy quantifies the uncertainty in the model’s probability distribution over possible classes. It gives more granularity than just the predicted class label by measuring “how confident” the prediction is.

Entropy is calculated from the model’s softmax probabilities:

H(p) = - Σ_k p_k log p_k

where p_k is the predicted probability for class k.

High predictive entropy indicates ambiguous predictions, which often correspond to edge or disputed cases. This helps prioritize which samples to investigate or flag for manual review.

Why Predictive Entropy Matters

Uncalibrated probability outputs are misleading: High-confidence incorrect predictions can cause costly mistakes.
Entropy detects uncertainty even when the top predicted class remains constant.
Entropy trends can reveal distribution shifts if the model encounters many samples with historically unseen feature patterns.

Edge Cases and Distribution Shift: The Root Causes of Dispute

Disputed cases often stem from edge cases and distribution shifts that violate training assumptions:

Edge cases: Samples that lie near decision boundaries or contain rare combinations of features, making predictions unstable.
Distribution shift: Changes in input data characteristics over time or across populations, causing the model to face unfamiliar contexts.

For example, in healthcare operations, a model trained on one demographic or enrollment period might underperform on emerging patient subgroups or during unknown rare events. These shifts manifest as increased disagreement between human and model labels, or rising predictive entropy.

Addressing Distribution Shift

The key is early detection and diagnosis. Alongside disagreement and entropy metrics, monitoring changes in input feature distributions and outcome likelihoods helps pinpoint when to enact retraining or specialized handling.

Data Gaps and Subgroup Coverage: Unequal Model Performance

We often see model performance disparities across subgroups—be it defined by race, language, age, or other demographics—stemming from data gaps. These gaps lead to:

Higher disagreement rates in underrepresented groups
Elevated predictive entropy signifying model uncertainty
Potentially unfair or biased outcomes violating compliance

It becomes vital to conduct subgroup-specific diagnostic experiments in Week 3, measuring disagreement rates and entropy stratified by subgroup. This reveals coverage holes and guides targeted data collection or model adjustments.

Objective Mismatch and Loss Function Tradeoffs

A subtle, often ignored cause behind disputed cases is the mismatch between training objectives and real-world decision criteria. Models typically optimize proxy losses (like cross-entropy) aiming to minimize overall error, but actual operational costs may weigh false positives and negatives differently.

Such objective mismatch manifests in:

Thresholds chosen “by feel,” obscuring true cost sensibles.
Failure to capture edge case risks in loss functions.
Disagreements arising due to misaligned incentives between model outputs and human decisions.

Discover more here

Hence, in diagnostic experiments, it’s critical to re-express thresholds and losses in terms of explicit cost models and perform controlled experiments to empirically measure impact on false positive/negative tradeoffs.

Controlled Experiments and Counterfactual Tests: Methodologies to Diagnose Disputes

We now arrive at the heart of diagnostic Week 3 activities — controlled experiments and counterfactual testing. These are rigorous tests designed to measure what happens when we intervene on disputed cases or suspect features.

Controlled Experiments

In these experiments, we manually categorize disputed samples into risk buckets informed by disagreement rate and predictive entropy, then apply customized treatments to evaluate performance improvements, such as:

Augmented training with inclusion of disputed labels
Re-weighting loss functions to prioritize edge cases
Human-in-the-loop mechanisms targeting high entropy or disagreement cases

Results guide whether re-training, threshold adjustments, or feature engineering are warranted.

Counterfactual Testing

This entails synthesizing or selecting samples that differ in controlled feature dimensions (counterfactuals) from disputed cases to understand the causal drivers behind disagreement.

For instance, in a lending risk model, altering income or employment history features in disputed cases and observing shifts in model predictions can spotlight sensitive attributes or proxy effects causing disagreements.

Augmented Training as a Remediation Strategy

The last mile is closing the loop with augmented training. This involves:

Incorporating disputed or ambiguous cases back into the training data, explicitly weighted to emphasize learning on these challenging points.
Data augmentation techniques to enlarge representation of edge cases and underrepresented subgroups.
Feature augmentation or enrichment to provide additional signal supporting better predictions.

Augmented training, informed by Week 3 diagnostic experiments, systematically enhances model robustness against distribution shifts and reduces disagreement with humans.

Summary and Best Practices

Week 3 diagnostic experiments for disputed cases are crucial for building trust and safety in decision systems. By leveraging disagreement rate and predictive entropy, we can spotlight the riskiest samples and subgroups with the greatest uncertainty.

Combining this signal with controlled experiments, counterfactual tests, and augmented training addresses root causes like data gaps, distribution shifts, and objective mismatch.

Things Accuracy Hides

Accuracy often masks model uncertainty and subgroup disparities.
Disagreement captures signal in what’s not yet well understood or modeled.
Thresholds tied to operational costs prevent hand-wavy “AI will handle it” claims.
Calibration and human-machine concordance are essential for safe deployment.

On the worst day in production? These measures become your canary in the coal mine, alerting you before systemic failures cascade.

References & Tools

Measuring Uncertainty in Deep Learning with Predictive Entropy
Fairness under Distributional Shift
Disagreement Rate in Ensemble Models

If you found this helpful, stay tuned for next predictive entropy week’s post on monitoring ensemble rollouts and retraining triggers!