An Arizona software issue reportedly kept inmates locked up when they should have been released, and it shows the real human cost of tech errors

  • A software issue caused Arizona prison inmates to be stuck behind bars longer than legally allowed.
  • A separate software issue resulted in potentially thousands of inmates wrongly held in max custody.
  • Other software issues have led to fatal outcomes, leading experts to call for increased oversight.
  • See more stories on Insider’s business page.

It’s easy to forget just how much of the modern world is powered by software, from trains and planes to emails and file transfers. Corene Kendrick, deputy director of the American Civil Liberties Union (ACLU)’s National Prison Project, says that when software is “working behind the scenes, that kind of business operations stuff, you just assume normally it’s doing its job.”

In fact, you may only notice it when things have gone wrong. Sometimes, that means a late email, or a corrupted file in cloud storage. But other times, software errors can have a startling human cost.

A recent whistleblower report from KJZZ found that “hundreds” of Arizona prison inmates who may be qualified for early release are being held in custody because its custom software for managing inmates — built at a cost of $24 million — can’t interpret a new state sentencing law. According to the whistleblowers, the Arizona Department of Corrections Rehabilitation & Reentry (ADCRR) has known about the issue since 2019.

“I can’t remember ever hearing of something just so systemic and profound affecting thousands of people with what we’re hearing here,” Kendrick told Insider.

This incident, and recent ones like it, are part of what experts call a troubling trend of poorly managed or designed tech within the criminal justice system and other public services, whether rooted in systematic bias like algorithmic risk assessments, or software that simply can’t interpret existing rules and laws.

The evidence suggests that when software is built for institutions like prisons or social services like welfare, the consequences of IT issues are devastating, with society’s most vulnerable often ending up bearing the cost. Experts say that oversight over government enterprise technology is long overdue — but at the same time, legislation must be handled with care for fear of exacerbating the issues.

“Those are all key elements here especially when we’re talking about people’s lives,” Forrester vice president Joseph Blankenship told Insider. “That’s what makes these stories so painful.” 

Arizona’s $24 million prison software has had other issues

The software in question, called Arizona Correctional Information System (ACIS), was created by ADCRR in partnership with IT company Business & Decision North America (BNDNA), and put into use in late 2019 after records show a contract for “development, implementation and maintenance services for Enterprise Services Platform” that began in 2013.

This most recent case isn’t the first time that ADCRR has found itself under scrutiny: ACLU Arizona and other legal advocacy groups filed a class-action lawsuit against ADCRR in 2012 for failing to meet inmates’ health care needs.

The case that followed resulted in Arizona officials admitting in court documents that ACIS couldn’t compute the amount of time inmates should spend in maximum custody, or housing in a single or double cell environment with limited movement and frequent monitoring  — essentially solitary confinement without the label, advocates have charged. The maximum custody issue with the software has not previously been reported.

The result was that inmates who should have been moved into less restrictive custody levels weren’t being transferred, despite being eligible. “What we found in monitoring the case and talking to incarcerated people is that folks were spending years and years stuck at the same level,” Kendrick, then a lead lawyer on the case with the Prison Law Office, said.

Kendrick said she estimated between 2,000 and 3,000 inmates are held at the maximum custody level at any given time and potentially impacted by the software issue; according to the Arizona corrections department, 1,851 people are currently in permanent maximum custody. 

Court documents from the lawsuit viewed by Insider suggest that the issues with inmates being denied early release described in the KJZZ report occurred approximately alongside the maximum custody issues.

A screenshot of the software previously used in Arizona prisons, taken from court documents, which the new custom software replaced.US District Court of Arizona

“These people have been stuck in very horrible conditions of maximum custody, not allowed very many privileges at all or out of cell time, and it’s just because they have no way of knowing this person is up for their 60 day review or their 30 day review,” Kendrick said. “Given what we know about the release dates and the max custody, it’s pretty unsettling that they seem to just have such a poor system for tracking their population.”

ADCRR spokesperson Bill Lamoreaux told Insider in a statement that “each sentence is manually reviewed and certified by our staff for accuracy prior to release, and the software was “never intended to automatically calculate this sentence modification.”

“Procedures have been implemented and remain in place to address this and any future statutory requirements,” he said, but did not comment on the maximum custody issue specifically.

BNDNA told Insider in a statement that its software is “working optimally based on the criteria provided by our customer,” and “we stand by the accuracy of our software and remain ready to make any and all adjustments to our software upon customer request.” Meanwhile, on its website, BNDNA claims its prison management software has a “successful track record of accurately calculating over 600,000 sentences.”

It’s also worth noting that the same software at the root of the controversy in Arizona is also used in Maryland prisons. A spokesperson for the Maryland Department of Public Safety and Correctional Services (DPSCS) spokesperson said it began using the software in November 2010, and the four software modules it currently uses are “working as expected.”

Just scratching the surface in software issues causing human impact

The tech issues in Arizona are the latest in a string of similar examples, showing that software problems can have a disastrous real-world impact on peoples’ lives.

In December 2015, a software glitch, previously reported by Insider, resulted in the incorrect early release of over 3,000 inmates in Washington state prisons over the course of 13 years. Worse, the Department of Corrections had been aware of the error for at least three years but had been unable to resolve it. The DOC was ultimately forced to track down prisoners who had been released early and return them to prison to complete their sentences.

Software issues can also have harmful consequences outside of prison software.

The Australian government used a software program called Centrelink, or “robodebt,” that automated the calculation and recovery of overpayments on welfare. Between 2015 to 2019, it wrongfully issued 470,000 debts, according to The Guardian, because a system error incorrectly calculated and compared recipients’ reported income. The results were tragic: Some who believed they owed money committed suicide.

Advocates say it’s time for more oversight of tech for public services

Advocates say that the entire issue goes beyond specific software glitches, and show there’s a need for better policy and more direct oversight in how these tools get deployed and used.

“There’s no independent entity that oversees, no independent inspector general that does investigation,” Kendrick said of Arizona’s corrections system — also noting that whatever processes there are for oversight over the state’s prisons don’t specify checking on the software that they use. 

Advocates also told Insider that in their view, ACIS sentencing issues weren’t just the fault of the software — ADCRR allowed it to persist.

“It’s not a software glitch,” Caroline Issacs, program director for the American Friends Service Committee, a Quaker organization that works on criminal justice reform, told Insider. “It’s a massive failure that was purposely ignored for two years. A software glitch is when something goes wrong and you get it back online. This is negligence on the part of the Department of Corrections.”

Lyria Bennett Moses, a law professor at University of New South Wales, told Insider the Australian robodebt disaster shows how software can be dangerous in real life because there are no “integrity checks” to make sure it complies with laws.

“As we move to a world where most legislation is implemented in software, we need to really think about how we ensure the integrity of the process,” Moses said. “The broader point is until we come up with something like that, we’re going to keep getting the same problem. Software providers are not accountable to the public in the same way that governments are.”

A closer collaboration with technologists might be an answer

A worldwide movement called Rules As Code, part of a larger push towards civic technology, offers one solution to public technology issues: When lawmakers draft legislation, they can simultaneously work with software engineers to write the software rules and make sure the code will be in compliance.

It also means that governments could take more responsibility for the technology, and software providers would have to fix bugs within a certain amount of time or frequently test their code. Lawmakers could also draft legislation that includes requirements for automation: “They can’t say, ‘Oh it’s a software provider, it’s not our fault,” said Moses.

Government software could also be released as open source, Moses said, making it so advocacy groups and developers can easily view the code, catch bugs, and get them fixed. Open Prison Education, for example, is an open source project — while not government funded or sanctioned — that aims to make it easier for inmates to get an education by enabling their laptops to sync with instructional material and make it viewable offline.

Regardless of what it ultimately looks like, experts agreed that legislation needs to be implemented with care, and increasingly, with the help of technology and the tech industry — necessitating a whole new layer of oversight and accountability for any systems that impact society.

Evonne Silva, senior director of criminal justice at Code for America, a civic tech group focused on local governments, told Insider that “many well-meaning laws falter because of failure in implementation.” “That not only undermines the intent of these laws, it erodes the public’s trust in government,” she said.

Source: Read Full Article