Artificial intelligence and algorithmic tools used by central government are to be published on a public register after warnings they can contain “entrenched” racism and bias.

Officials confirmed this weekend that tools challenged by campaigners over alleged secrecy and a risk of bias will be named shortly. The technology has been used for a range of purposes, from trying to detect sham marriages to rooting out fraud and error in benefit claims.

The move is a victory for campaigners who have been challenging the deployment of AI in central government in advance of what is likely to be a rapid rollout of the technology in the public sector. Caroline Selman, a senior research fellow at the Public Law Project (PLP), an access-to-justice charity, stated, “We need to make sure public bodies are publishing the information about these tools, which are being rapidly rolled out. It is in everyone’s interest that the technology which is adopted is lawful, fair and non-discriminatory.”

Previous Concerns Over AI Algorithms

In August 2020, the Home Office agreed to stop using a computer algorithm to help sort visa applications after it was claimed it contained “entrenched racism and bias”. Officials suspended the algorithm after a legal challenge by the Joint Council for the Welfare of Immigrants and the digital rights group Foxglove.

Foxglove claimed that some nationalities were automatically given a “red” traffic-light risk score, leading to a higher likelihood of visa denial. This process was viewed as racial discrimination.

Challenges with Sham Marriage Detection

The department was also challenged last year over an algorithmic tool to detect sham marriages used to subvert immigration controls. The PLP noted that it appeared it could discriminate against individuals from certain countries, with an equality assessment revealing that Bulgarian, Greek, Romanian, and Albanian individuals were more likely to be referred for investigation.

Government Warnings on AI Bias

The government’s Centre for Data Ethics and Innovation, now the Responsible Technology Adoption Unit, warned in a report in November 2020 that there were numerous examples where the new technology had “entrenched or amplified historic biases, or even created new forms of bias or unfairness”.

The centre helped develop an algorithmic transparency recording standard in November 2021 for public bodies deploying AI and algorithmic tools. It proposed that models which interact with the public or significantly influence decisions be published on a register or “repository”, detailing how and why they were being used.

Current Status of AI Transparency

To date, just nine records have been published in three years on the repository. None of the models is operated by the Home Office or Department for Work and Pensions (DWP), which have operated some of the most controversial systems.

The last government stated in a consultation response on AI regulation in February that departments would be mandated to comply with the reporting standard. The Department for Science, Innovation and Technology (DSIT) confirmed this weekend that departments would now report on the use of the technology under the standard.

A DSIT spokesperson remarked, “Technology has huge potential to improve public services, but we know it’s important to maintain the right safeguards including, where appropriate, human oversight and other forms of governance.”

Future Implications

Departments are likely to face further calls to reveal more details on how their AI systems work and the measures taken to reduce the risk of bias. The DWP is using AI to detect potential fraud in advance claims for universal credit, with more in development to detect fraud in other areas.

In its latest annual report, it claims to have conducted a “fairness” analysis on its use of AI for universal credit advance claims, which did not “present any immediate concerns of discrimination”. However, the DWP has not provided any details of its assessment due to concerns that publication could “allow fraudsters to understand how the model operates”.

The PLP is supporting possible legal action against the DWP over the use of the technology, pressing the department for details on how it is being used and the measures taken to mitigate harm. The project has compiled its own register of automated decision-making tools in government, with 55 tools tracked to date.