CATHY O’NEIL: Look Who’s Fighting Our Algorithmic Overlords.

Consider Themis, a new, open-source bias detection tool developed by computer scientists at the University of Massachusetts Amherst. It tests “black box” algorithms by feeding them inputs with slight differences and seeing what comes out — much as sociologists have tested companies’ hiring practices by sending them resumes with white-sounding and black-sounding names. This can be valuable in understanding whether an algorithm is fundamentally flawed.

The software, however, has a key limitation: It changes just one attribute at a time. To quantify the difference between white and black candidates, it must assume that they are identical in every other way. But in real life, whites and blacks, or men and women, tend to differ systematically in many ways — data points that algorithms can lock onto even with no information on race or gender. How many white engineers matriculated from Howard University? What are the chances that a woman attended high-school math camp?

Untangling cultural bias from actual differences in qualifications isn’t easy.

Not only am I certain that culture is an actual difference, I’d argue that it is the defining difference between individuals — and that the brains behind Themis are trying to pry open the black boxes in a spectacularly unhelpful way.