Many people are nervous about the prospect of using algorithms to predict crime, and a legal battle in Wisconsin is illustrating why. The state’s Supreme Court is close to ruling on an appeal from Eric Loomis, who claims that the justice system relied too heavily on its Compas algorithm to determine the likelihood of repeat offenses and sentence him to 6 years in prison. His attorneys claim that the code is “full of holes,” including secret criteria and generic decisions that aren’t as individually tailored as they have to be. For instance, they’ll skew predictions based on your gender or age — how does that reflect the actual offender?

Algorithms in sentencing aren’t new. They’ve been in use for over 10 years, and their deployment is widening to states like Pennsylvania. However, the court challenge could force Wisconsin and other states to think about the weight they give to algorithms. While they do hold the promise of both preventing repeat offenses and avoiding excessive sentences for low-threat criminals, the American Civil Liberties Union is worried that they can amplify biases or make mistakes based on imperfect law enforcement data. Without transparency, it’s hard to say for sure that Loomis and other convicts are getting an appropriate amount of prison time.