Definition
Fairness metrics are quantitative benchmarks used to detect and mitigate algorithmic bias in RAG systems and AI agents, often involving a trade-off between strict demographic parity and raw retrieval precision. They evaluate whether LLM outputs or retrieved documents exhibit disparate impact, stereotyping, or exclusion across different demographic groups or protected attributes.
Distinguishes social and demographic equity from technical performance metrics like latency or F1 score.
"An Auditor's Prism: A single beam of data passes through and is split into demographic spectra to ensure no single color is dimmed or distorted."
- Retrieval Bias(Component)
- Demographic Parity(Prerequisite)
- LLM-as-a-Judge(Prerequisite)
- Stereotyping Detection(Component)
Conceptual Overview
Fairness metrics are quantitative benchmarks used to detect and mitigate algorithmic bias in RAG systems and AI agents, often involving a trade-off between strict demographic parity and raw retrieval precision. They evaluate whether LLM outputs or retrieved documents exhibit disparate impact, stereotyping, or exclusion across different demographic groups or protected attributes.
Disambiguation
Distinguishes social and demographic equity from technical performance metrics like latency or F1 score.
Visual Analog
An Auditor's Prism: A single beam of data passes through and is split into demographic spectra to ensure no single color is dimmed or distorted.