Algorithmic bias is a systematic difference in how an AI system treats or represents groups of people, such as by gender, ethnicity, age, disability or dialect, that the task does not justify. It is a property of a whole system in use, not just of a model: the same model can be fair in one application and discriminatory in another.

It is a different thing from the statistical “bias” in the bias-variance tradeoff , which describes a model’s systematic error on the data in general rather than unequal treatment of groups.

Where it comes from

  • Training data. A model learns the patterns in its data, including historical discrimination and under-representation. If most images labelled “CEO” show white men, an image generator will too.
  • Labels and targets. Choosing what to predict can encode bias: predicting “who was hired before” reproduces past hiring decisions.
  • Design choices. Prompt templates, ranking order, thresholds and post-processing all shape outcomes. One 2025 study of 22 LLMs found the first-listed candidate was chosen 63.5% of the time, a bias created by input order alone.
  • Vendor changes. Hosted models are updated, and some providers rewrite prompts before generation. Results can change without any change on your side.
  • Use in context. A model used outside the population or purpose it was tested for can behave differently.

How it is measured

There is no single fairness number. Common approaches:

  • Representation audits: generate or retrieve many outputs for neutral prompts and count who appears, compared with a real-world baseline such as labour statistics.
  • Paired or matched tests: submit inputs that differ only in a group signal, such as a name or dialect, and compare outcomes. This is the method used in hiring audits.
  • Outcome metrics: compare rates of positive decisions, error rates or calibration across groups. Several of these definitions cannot all be satisfied at once, so choosing one is a decision about values, not only engineering.

Results depend heavily on the model generation, prompt wording and measure chosen. Studies of LLM resume screening have found pro-White and pro-male preferences in some systems and the reverse in newer ones. See AI bias in numbers, 2026 for current figures with their sources.

What to do about it

  • Define which groups and harms matter for your use case before choosing a metric.
  • Test by group before launch and after every model, prompt or vendor change.
  • Keep a human accountable for consequential decisions about people, and document the reasoning; this is also what automated decision-making rules under GDPR expect.
  • In the EU, AI used in hiring, worker management, credit and other Annex III areas is high-risk under the AI Act, with obligations applying from 2 December 2027.

Sources

Further reading