Notes on AI Bias

This, hugely simplified, is the ‘AI bias’ or ‘machine learning bias’ problem: a system for finding patterns in data might find the wrong patterns, and you might not realise. A machine learning system that is very good at spotting skin cancer on pale skin might be worse at spotting skin cancer on darker coloured skin, or vice versa, not perhaps because of bias in the sample but because you might need to construct the model differently to begin with to pick out different characteristics. One could easily imagine a system that looks at the historical data and learns to associate ‘female’ first names with lower risk, so you would remove the first names from the data – but, as with the Amazon example above, there might be other factors that reveal the gender to the system (though of course it would have no concept of gender, or indeed cars), and you might not realise this until the regulator did an ex ante statistical analysis of the quotes you’ve given and fined you.

Source: www.ben-evans.com