Covert Dialect Bias in LLMs

This is my Master’s thesis project, supervised by Dr. Rita Orji at the Persuasive Computing Lab, Faculty of Computer Science, Dalhousie University.

The project measures covert dialect prejudice in large language models across housing-related social judgments, employing a computational matched-guise methodology adapted from sociolinguistics and social psychology, where the technique was originally developed to measure listeners’ implicit attitudes toward speakers of different language varieties. Rather than relying on explicit references to racial or ethnic identity, the work investigates how implicit cues in a prompt — such as dialect and phrasing — can steer a model’s social judgments. We characterize this dialect-based attribute inference as a contextual integrity concern, in Helen Nissenbaum’s sense of the term: information about a speaker’s likely identity is inferred and acted upon in a context where that inference was never volunteered.

Phase 1 of this work was accepted at the 9th AAAI/ACM Conference on AI, Ethics, and Society (AIES 2026) (Abdullah & Orji, 2026). Phase 2 extends the framework to study contextual integrity violations more broadly.

The broader phenomenon this work engages with has also been documented at scale by Hofmann et al. (2024), who show that language models can express covertly prejudiced decisions about speakers on the basis of dialect alone.

Related work on this site: Narrlytics on making recommender behaviour legible to the reader, and Persuasive Strategies for Mindful use of AI on the human side of generative AI use. See also my full list of publications.