Icahn School of Medicine

AEquity: A Deep Learning Based Metric for Detecting, Characterizing and Mitigating Dataset Bias

Diagnostic and prognostic algorithms can recapitulate and perpetuate systemic biases against underserved populations. Post-hoc technical solutions do not work well because they are unable to overcome biased data used to train algorithms.. A more data-centric approach may help address bias earlier in the process of development of algorithms. We present AEquity (AEq), a sample-efficient deep-learning based metric, that measures the learnability of subsets of data representing underserved populations. We then show how the systematic analysis of AEq values across subpopulations allows the identification of different manifestations of bias in two healthcare datasets with demonstrated bias. In the first case, we analyze a computer vision model trained on chest radiographs to investigate its underperformance on chest X-rays from Black individuals for various diagnoses and develop targeted interventions at the dataset level which mitigate bias. We also apply AEq to a cost-predictive model to show how the choice of the label leads to bias against black patients, which can be remedied by predicting comorbidities instead of costs. Thus, AEquity is a novel and broadly applicable metric which can be applied to diagnose and remediate bias in healthcare datasets.