FAIR-01 · Fairness

Algorithm Bias

A hospital trains a model to flag patients at risk from a blood marker and a symptom score. In group B, the marker reads lower for the same illness, much as some real devices read differently across patients. If the training data is mostly group A, the model learns group A's pattern.

The training data synthetic

Group B's share of training data How much lower B's marker reads

0 means the marker behaves the same in both groups.

Flag patients above risk

Lower catches more sick patients but raises false alarms.

Let the model account for the group

Gives the model the group as an input so it can learn B's offset. It fixes the numbers here, but using group membership raises its own questions.

Model

Test patients, 600 per group filled: sick · hollow: healthy

Sick patients caught sensitivity

Healthy patients flagged false alarms

Bias without intent

Nobody told this model to treat the groups differently. It learned one decision line from data that was mostly group A, and that line sits in the wrong place for group B.

Overall accuracy hides it

Because group A dominates, the model can look accurate overall while missing many sick patients in group B. Fairness checks report error rates for each group separately.

It happens in practice

A widely used US care-management algorithm underestimated Black patients' needs because it predicted cost rather than illness (Obermeyer et al., Science, 2019). Pulse oximeters missed low oxygen more often in Black patients (Sjoding et al., NEJM, 2020).

All patients here are synthetic: generated in your browser from made-up distributions. The model is a real logistic regression trained by gradient descent each time you change a setting. Sources: Obermeyer et al., Science (2019); Sjoding et al., NEJM (2020). Educational only; not medical advice.