What is Bias?
Text-to-image systems turn trans people into caricatures, translation systems obliterate neo-pronouns and word embeddings place queer identity terms close to negative adjectives. All of these examples are all manifestations of what we call bias. But while we can more or less agree that all of the previous examples fall under the same category, NLP papers have a hard time defining the term, which in turn leads to difficulties in comparing approaches. How can one call a system more or less biased, when the how and what of measuring bias is never actually stated?
In their 2020 paper Blodgett et al. review 146 NLP papers that are concerned with bias and fairness. In doing so they discover that in 23 of them the nature of the bias in question is not even mentioned. So how can we move beyond a vague “the system does something bad and we want to change it”? The first step is to provide us with language. Blodgett et al. suggest the following categories:
Allocational harms: The harms done by a system that allocates resources unfairly between different groups. Think of a job ad platform that displays more ads for leadership roles to men than to the other genders.
Representational harms: The harms done by a system that represents a group in a less favourable light or ignores their existence altogether. Think of an image tagging tool that mislabels images showing people of colour.
That last category of harms can be subdivided even further into stereotyping, which propagates negative beliefs about groups of people, and differences in system performance for different groups of users, for example a speech-transcription system that makes more errors when transcribing African American speakers.
With these terms we can already make more precise statements about what bias exactly a research question is examining. But a shared vocabulary is only the first step. Building on their review Blodgett et al. provide 3 recommendations to researchers tackling the problem of bias:
1 - Read (and cite!) literature outside of NLP
Coming from a machine learning perspective it is easy to see language as yet another signal that can be handled and predicted with the same tools one would use to predict the maintenance needs of a steel press. But language is a different beast altogether - it constructs and maintains social hierarchies. Through language groups are labelled, beliefs about the groups are transmitted and social inequalities reinforced. Language is also used in the counter movement to this, searching to remove stigmatising descriptions e.g. the term “illegal” for immigrants. Many queer identity terms (including “queer”) have gone through cycles of being used as insults before being reclaimed by the community and eventually adapted into “standard” language in a neutral or positive meaning.
What is considered “standard” is the second important question when working on the topic of bias and language. Whose language practice is considered correct and educated and whose is considered offensive? Whose text is considered as “noisy” and is therefore filtered from training data? Blodget et al. suggest that researchers examine these questions at each step of the development cycle starting with the task definition, the data and the evaluation. What are the assumptions we make about potential users? How are data sets collected? Are there qualitative as well as quantitative ways to evaluate results? And most of all - are you measuring the most concerning harms or just the ones that are easy to measure?
2 - Provide explicit statements of why the system behaviours that are described as “bias” are harmful, in what ways, and to whom
Bias is not self-evident and vagueness makes it hard to compare and combine existing approaches to the problem. Claiming to treat bias but then shying away from clearly articulating what is deemed good and bad is therefore a self-defeating endeavour. Research that examines bias makes claims of norms and values, but it might seem very counterintuitive to someone dealing in system performance numbers and loss functions to make explicit statements about that. Blodgett et al. suggest taking the relation of a system to existing hierarchies as a start point. Some systems might be harmful because they uphold existing hierarchies, e.g. by assigning queer identity terms negative sentiment values. A researcher should always ask themselves what kind of system behaviours exactly are flagged as bias, where these behaviours stem from and who exactly they harm.
3 - Engage with the lived experience of system users
Machine learning experts love to chop down real world problems into development, test and training data sets and to throw hands full of different model architectures at them until one sticks. It is our superpower, but in terms of bias research this approach might often be misguided. Language use does not happen in challenge data sets conveniently published on benchmarking sites. It happens in the real world and engaging in it can be costly and time intensive, two of the least favourite words of any machine learning practitioner. Moreover this perspective forces us to interrogate our own role in upholding social hierarchies. Should the system that we are working on even exist? Should crucial development decisions even be in the hand of technologists (aka, us) rather than in the hands of the affected communities? Research domains outside of NLP can provide bule-prints on how to navigate this new territory. Value-sensitive design and participatory research provide guidelines on how to better involve stakeholders in research and design of NLP systems.
In a world where power over machine learning systems is concentrated in the hands of a few massive companies, research on the negative impacts of these systems is more needed than ever. Blodgett et al.s recommendations should not be read as discouragement - quite on the contrary. Only when we know what we talk about, when we talk about bias, can we move the field forward.
This post was written by Sabine Weber. Sabine is a queer person who finished their PhD at the University of Edinburgh. They are interested in multilingual NLP, AI ethics, science communication and art. They organized Queer in AI socials and are D&I Chair of EACL 2024. You can find them on twitter as @multilingual_s or on their website.