Ghosts in the Machines

Introduction

A machine can’t be biased? It follows directions. It is objective—governed by data and code! Right?! Well, not really. A computer, for example, processes information. It receives inputs (like clicks on your keyboard) and generates outputs. Algorithms are a set of instructions for carrying out that process. Algorithms are also the building blocks of new technologies, like generative artificial intelligence (GenAI). AI models use complex algorithms to simulate advanced reasoning. The creators of these complex algorithms encode step by step processes, directing the algorithm to handle data in a certain way. As AI becomes increasingly widespread in daily life, the underlying algorithms warrant scrutiny.

Algorithms—because they are coded by humans—and AI models—because they take in massive amounts of human data to “learn”—are not objective. Some researchers have called these models WEIRD—Western, Educated, Industrialized, Rich, and Democratic—to demonstrate the lack of representation of the global south and the developing world in AI development and design. Algorithmic bias occurs when systemic errors or biases encoded in algorithms produce discriminatory outcomes. From loans to language, we have looked at the ways algorithms can harm marginalized communities and reviewed mitigation strategies that can be used to exorcise these ghosts in the machines.

Where does the bias come from?

Data is often referred to as the “oil of the digital age“, powering technological advancements. However, just as oil quality affects machine performance, the quality of data significantly impacts algorithmic outputs. Algorithms learn from the data on which they are trained. If this data is biased, incomplete, or inaccurate, the outputs will also be biased. For instance, a CV-screening algorithm trained on a biased dataset might favor certain demographics over others, even if the algorithm’s instructions are neutral.

In addition to data used to train a model potentially encoding bias, those responsible for building the model can also consciously or unconsciously encode their own prejudices. For example, a developer may unfairly weigh certain characteristics (e.g., income, education) based on beliefs they hold, leading to discriminatory algorithmic outcomes affecting marginalized communities.

What does algorithmic bias look like?

Racial bias 

In the United States, where racial inequality is baked into many public and private systems, algorithms can reproduce racial inequalities at a grand scale. For example, there is overwhelming evidence that lending algorithms, pre-coded systems that help creditors decide who to lend money to, are biased against minority groups. This impacts the economic mobility of vulnerable communities and erodes the ability of marginalized people to pursue opportunities like small-business ventures and tertiary education. Because algorithms can be trained to correlate risk to certain factors (e.g., zip codes and educational attainment), they lift up individuals who have historic access to privilege and suppress those who do not. 

Policing is another area where algorithms can reproduce bias. Predictive Policing is a law enforcement strategy employing an algorithm to determine where crime is likely to occur and who is at a higher risk of perpetrating it. The data relied upon to train these algorithms is skewed. The training data often contains arrest statistics from police departments, which are not indicative of a crime being committed. In the United States, Black people are more likely to have run-ins with police, and algorithms themselves are not capable of understanding the historical and socioeconomic reasons why this discrepancy exists. As algorithms take their biased data to make biased predictions of where crime is likely to occur, a UK study suggests these algorithms prime officers to expect trouble. This priming makes law enforcement more likely to stop people because of bias, rather than need. Policing algorithms, in many ways, are self-fulfilling prophecies. They take police data—a skewed, historical picture of crime in a jurisdiction—and use it to identify crime “hotspots” and likely criminals. Overzealous officers are sent to monitor these areas and these people, which are disproportionately people of color. 

Institutions in the United States are not alone in their embrace of algorithms. In the European Union, many member states are employing frontier technologies to secure their borders. These technologies increase the likelihood that the human rights of migrants will be violated. AI technologies are present throughout various stages of migration, from administrative processing to automated border surveillance. AI making preliminary decisions about a migrant based on their “risky” characteristics and the gathering of biometric data into a black box with little public oversight are just two of the ways algorithms create grave challenges at borders. Researchers and human rights advocates have also identified a “funnel effect”, where migrants increasingly take alternative routes to avoid encountering technologically advanced borders. These routes are typically more dangerous, sometimes resulting in severe injury or death. As the technologies powering these interventions develop faster than the regulations surrounding them, there is immense potential for harm and few protections in place. While AI can make border facilities more efficient, they can also inflict unjust harm on vulnerable people.

Linguistic bias

The current gap in language representation in AI models is vast. Researchers have highlighted how AI-based language technology is currently limited to 2 to 3 percent of the world’s most widely spoken and/or financially and politically best supported languages. This means that models are only able to understand and express concepts that belong to specific languages and cultures. This leads to techno-linguistic bias, widening existing inequalities, since the current approach to building multilingual models is mostly developed from the perspective of a single language and culture, usually English. As the American writer Rita Mae Brown said, “language is the roadmap of a culture” when we speak, there are underlying meanings to words, so having these “multilingual” models, based on a single perspective, leads to misunderstanding and incorrect outputs. This can further widen the digital divide, deepen inequalities, increase underrepresentation, while further discriminating against certain language groups. 

A more meaningful language representation is critical to develop models that can be applied globally.  This can also be beneficial to diversify the data set and include other contexts that can be enriching for the model’s performance. It can be done through collaboration with local groups, further democratizing the development of technology and making it more inclusive. The US-led resolution in the UN General Assembly on AI highlights the importance of promoting AI systems that preserve linguistic and cultural diversity. 

Mitigation

Increasingly complex and capable algorithms whose inner workings can be inscrutable have the potential to encode human biases and unleash them on an unprecedented scale. However, AI developers, companies, institutions, and people seeking to use them have a number of mitigations available to them.

Human Rights Impact Assessments

Human Rights Impact Assessments (HRIAs) are mandated by frameworks like the UNGPs, EU AI Act, and EU Digital Services Act. They are a crucial tool for proactively identifying, addressing, and mitigating human rights risks. This exercise is essential for developers, ensuring they consider potential human rights impacts throughout the development process. HRIAs are most effective when used as an ongoing, iterative process that is regularly updated as an algorithm’s deployment and use evolve.

Filtering training data

Data filtering is the process of excluding erroneous or harmful datapoints from the data to reduce the inclusion of problematic content wherever possible before training. This reduces the chance that a chatbot, for example, will respond with toxic speech or personal information.

Enforcement and expansion of AI regulation

Governments around the world are working to understand and regulate artificial intelligence, but they are struggling to keep up with a rapidly developing industry. Regulations that prohibit the use of AI models in certain fields based on the risk they pose, as the EU AI Act does, warrant expansion and broader enforcement. To protect the most vulnerable communities from prejudicial algorithms, governments should work to limit the use of AI in various sectors where it can wreak social havoc at scale. This does not mean a moratorium on AI development, but rather a measured approach to addressing the harmful impact of AI models. Here smart regulatory approaches are crucial. This includes the need for multi stakeholder regulatory development, including input from the AI industry, to inform practical and effective regulation.

Human oversight

When algorithms make decisions that will impact human rights, there should be human oversight to  assess the decision. EU law has codified these requirements in the GDPR and EU AI Act, determining that there should be human oversight, usually referred to as a “human in the loop” with adequate training and authority to fulfill the tasks. The EU AI Act even requires at least two persons to verify the decision of a biometric identification system that can cause significant consequences or risks. However, as algorithms become more complex and widespread, human oversight becomes increasingly challenging. It may become difficult for humans to fully understand and validate the output of highly sophisticated systems, and the challenge of monitoring AI outputs at scale presents a huge logistical challenge.

Complaint mechanisms

Complaint mechanisms help people challenge automated decisions, promoting transparency and accountability. This will allow individuals to understand the reasons behind a decision and seek recourse.

Accuracy rate

Accuracy rates provide valuable context for understanding the performance of algorithmic systems, indicating the percentage of outputs that may contain errors. The EU AI Act mandates the inclusion of accuracy rates in system instructions. However, it’s crucial to remember that these rates are estimates based on controlled testing and may not perfectly reflect real-world performance.

Post-market monitoring

Post-market monitoring is a test which assesses how a system is performing in the specific context in which it is deployed. The EU AI Act emphasizes the importance of post-market monitoring for high-risk AI systems. This involves continuous assessment of system performance, identification of unintended consequences, and gathering user feedback. By regularly monitoring and evaluating AI systems, organizations can promote responsible use and address potential risks promptly. The European Commission will adopt an implementing act with details and a template for the post-market monitoring plan by 2026, so the way this will be implemented remains to be established.

Conclusion

Algorithms are powerful tools that can supercharge human productivity and efficiency. They are also capable of reproducing and entrenching inequalities. There are methods for mitigating these adverse effects, but as regulatory regimes struggle to keep pace with the industry, algorithms continue to warrant added scrutiny. Nonetheless, AI companies, including startups, can be responsible developers of these transformative technologies. By assessing the impact of their products on human rights, filtering training data, having multiple human reviewers throughout product development, and monitoring the model once deployed, AI companies can more effectively mitigate risks in their models. As AI systems become ever-more complex and capable, the challenges of assessing and mitigating potential adverse impacts will become more urgent and challenging. Therefore, companies that build responsible AI approaches into their DNA will have a clear advantage. 

If you have any questions, please do not hesitate to contact us at team@weadapt.io.