Dehumanizing Machines: Mitigating Anthropomorphic Behaviors in Text Generation Systems
Myra Cheng, Su Lin Blodgett, Alicia DeVrio, Lisa Egede, Alexandra Olteanu
2025 Meeting of the Association for Computational Linguistics | July 2025
SAC Highlight
Return to Microsoft Research Lab – Montréal
FATE | Montréal
Publications
-
-
-
Understanding and Meeting Practitioner Needs When Measuring Representational Harms Caused by LLM-Based Systems
Emma Harvey, Emily Sheng, Su Lin Blodgett, Alex Chouldechova, Jean Garcia-Gathright, Alexandra Olteanu, Hanna Wallach
Findings of the Association for Computational Linguistics: ACL 2025 | July 2025
-
Measuring Machine Learning Harms from Stereotypes: Requires Understanding Who is Being Harmed by Which Errors in What Ways
Angelina Wang, Xuechunzi Bai, Solon Barocas, Su Lin Blodgett
2025 ACM Conference on Fairness, Accountability, and Transparency | June 2025
-
Rigor in AI: Doing Rigorous AI Work Requires a Broader, Responsible AI-Informed Conception of Rigor
Alexandra Olteanu, Su Lin Blodgett, Agathe Balayn, Angelina Wang, Fernando Diaz, Flavio du Pin Calmon, Margaret Mitchell, Michael Ekstrand, Reuben Binns, Solon Barocas
June 2025
-
Canvil: Designerly Adaptation for LLM-Powered User Experiences
K. Feng, Q. Vera Liao, Ziang Xiao, Jennifer Wortman Vaughan, Amy Zhang, David W. McDonald
CHI 2025 | April 2025
-
A Taxonomy of Linguistic Expressions That Contribute To Anthropomorphism of Language Technologies
Alicia DeVrio, Myra Cheng, Lisa Egede, Alexandra Olteanu, Su Lin Blodgett
CHI 2025 | April 2025
-
Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistencies
Sunnie S. Y. Kim, Jennifer Wortman Vaughan, Q. Vera Liao, Tania Lombrozo, Olga Russakovsky
CHI 2025 | April 2025
Honorable Mention, CHI 2025
-
«I Am the One and Only, Your Cyber BFF»: Understanding the Impact of GenAI Requires Understanding the Impact of Anthropomorphic AI
Myra Cheng, Alicia DeVrio, Lisa Egede, Su Lin Blodgett, Alexandra Olteanu
April 2025
-
Generation Probabilities Are Not Enough: Uncertainty Highlighting in AI Code Completions
Helena Vasconcelos, Gagan Bansal, Adam Fourney, Q. Vera Liao, Jennifer Wortman Vaughan
ToCHI | April 2025, Vol 32(1)
-
AI Automatons: AI Systems Intended to Imitate Humans
Alexandra Olteanu, Solon Barocas, Su Lin Blodgett, Lisa Egede, Alicia DeVrio, Myra Cheng
March 2025
-
Position: Evaluating Generative AI Systems is a Social Science Measurement Challenge
Hanna Wallach, Meera Desai, A. Feder Cooper, Angelina Wang, Chad Atalla, Solon Barocas, Su Lin Blodgett, Alex Chouldechova, Emily Corvi, Alex Dow, Jean Garcia-Gathright, Alexandra Olteanu, Nick Pangakis, Stefanie Reed, Emily Sheng, Dan Vann, Jennifer Wortman Vaughan, Matthew Vogel, Hannah Washington, Abigail Z. Jacobs
ICML 2025 | January 2025
-
-
-
Gaps Between Research and Practice When Measuring Representational Harms Caused by LLM-Based Systems
Emma Harvey, Emily Sheng, Su Lin Blodgett, Alex Chouldechova, Jean Garcia-Gathright, Alexandra Olteanu, Hanna Wallach
November 2024
-
Evaluating Generative AI Systems is a Social Science Measurement Challenge
Hanna Wallach, Meera Desai, Nick Pangakis, A. Feder Cooper, Angelina Wang, Solon Barocas, Alex Chouldechova, Chad Atalla, Su Lin Blodgett, Emily Corvi, Alex Dow, Jean Garcia-Gathright, Alexandra Olteanu, Stefanie Reed, Emily Sheng, Dan Vann, Jennifer Wortman Vaughan, Matthew Vogel, Hannah Washington, Abigail Z. Jacobs
November 2024
-
Metrics for What, Metrics for Whom: Assessing Actionability of Bias Evaluation Metrics in NLP
Pieter Delobelle, Giuseppe Attanasio, Debora Nozza, Su Lin Blodgett, Zeerak Talat
2024 Empirical Methods in Natural Language Processing | November 2024
-
«I’m Not Sure, But…»: Examining the Impact of Large Language Models’ Uncertainty Expression on User Reliance and Trust
Sunnie S. Y. Kim, Q. Vera Liao, Mihaela Vorvoreanu, Stephanie Ballard, Jennifer Wortman Vaughan
7th ACM Conference on Fairness, Accountability, and Transparency (FAccT 2024) | June 2024
-
The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels
Eve Fleisig, Su Lin Blodgett, Dan Klein, Zeerak Talat
NAACL 2024 | June 2024
-
“One-Size-Fits-All”? Examining Expectations around What Constitute “Fair” or “Good” NLG System Behaviors
Li Lucy, Su Lin Blodgett, Milad Shokouhi, Hanna Wallach, Alexandra Olteanu
NAACL 2024 | June 2024
-
AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap
Q. Vera Liao, Jennifer Wortman Vaughan
Harvard Data Science Review | February 2024
-
-
-
Responsible AI Considerations in Text Summarization Research: A Review of Current Practices
Yu Lu Liu, Meng Cao, Su Lin Blodgett, Jackie Chi Kit Cheung, Alexandra Olteanu, Adam Trischler
Findings of EMNLP 2023 | December 2023
-
Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations
Valerie Chen, Q. Vera Liao, Jennifer Wortman Vaughan, Gagan Bansal
Proc. ACM Hum.-Comput. Interact. | October 2023, Vol 7
Article 370
-
FairPrism: Evaluating Fairness-Related Harms in Text Generation
Eve Fleisig, Aubrie Amstutz, Chad Atalla, Su Lin Blodgett, Hal Daumé III, Alexandra Olteanu, Emily Sheng, Dan Vann, Hanna Wallach
ACL 2023 | July 2023
-
This Prompt is Measuring : Evaluating Bias Evaluation in Language Models
Seraphina Goldfarb-Tarrant, Eddie Ungless, Esma Balkir, Su Lin Blodgett
Findings of the Association for Computational Linguistics: ACL 2023 | July 2023
-
AHA! Facilitating AI Impact Assessment by Generating Examples of Harms
Zana Buçinca, Chau Minh Pham, Maurice Jakesch, Marco Tulio Ribeiro, Alexandra Olteanu, Saleema Amershi
June 2023
-
Can Workers Meaningfully Consent To Workplace Wellbeing Technologies?
Shreya Chowdhary, Anna Kawakami, Mary L. Gray, Jina Suh, Alexandra Olteanu, Koustuv Saha
2023 ACM Conference on Fairness, Accountability, and Transparency | June 2023
-
The Kitmus Test: Evaluating Knowledge Integration From Multiple Sources In Natural Language Understanding Systems
Akshatha Arodi, Martin Pömsl, Kaheer Suleman, Adam Trischler, Alexandra Olteanu, Jackie Chi Kit Cheung
ACL 2023 | May 2023
-
Designerly Understanding: Information Needs for Model Transparency to Support Design Ideation for AI-Powered User Experience
Q. Vera Liao, Hariharan Subramonyam, Jennifer Wang, Jennifer Wortman Vaughan
2023 ACM CHI Conference on Human Factors in Computing Systems (CHI 2023) | April 2023
-
fAIlureNotes: Supporting Designers in Understanding the Limits of AI Models for Computer Vision Tasks
Steven Moore, Q. Vera Liao, Hariharan Subramonyam
CHI 2023 | April 2023
-
Sensing Wellbeing in the Workplace, Why and For Whom? Envisioning Impacts with Organizational Stakeholders
Anna Kawakami, Shreya Chowdhary, Shamsi T. Iqbal, Q. Liao, Alexandra Olteanu, J. Suh, Koustuv Saha, Alexandra Olteanu, Jina Suh
Proceedings of the ACM on Human-Computer Interaction | March 2023, Vol 7: pp. 1-33
-
Powering an AI Chatbot with Expert Sourcing to Support Credible Information Access
Ziang Xiao, Q. Vera Liao, Michelle X. Zhou, Tyrone Grandison, Yunyao Li
IUI 2023 | March 2023
-
Taxonomizing and Measuring Representational Harms: A Look at Image Tagging
Jared Katzman, Angelina Wang, Morgan Scheuerman, Su Lin Blodgett, Kristen Laird, Hanna Wallach, Solon Barocas
AAAI 2022 | February 2023
-
-
-
Generation Probabilities are Not Enough: Improving Error Highlighting for AI Code Suggestions
Helena Vasconcelos, Gagan Bansal, Adam Fourney, Q. Vera Liao, Jennifer Wortman Vaughan
NeurIPS Workshop on Human-Centered AI | October 2022
-
Connecting Algorithmic Research and Usage Contexts: A Perspective of Contextualized Evaluation for Explainable AI
Q. Vera Liao, Yunfeng Zhang, Ronny Luss, Finale Doshi-Velez, Amit Dhurandhar
AAAI Conference on Human Computation and Crowdsourcing | October 2022
Best Paper Award, Honorable Mention
-
Examining Responsibility and Deliberation in AI Impact Statements and Ethics Reviews
David Liu, Priyanka Nanayakkara, Sarah Ariyan Sakha, Grace Abuhamad, Su Lin Blodgett, Nicholas Diakopoulos, Jessica R. Hullman, Tina Eliassi-Rad
AIES 2022 | July 2022
-
How Different Groups Prioritize Ethical Values for Responsible AI
Maurice Jakesch, Zana Buçinca, Saleema Amershi, Alexandra Olteanu
FAccT 2022 | May 2022
-
Deconstructing NLG Evaluation: Evaluation Practices, Assumptions, and Their Implications
Kaitlyn Zhou, Su Lin Blodgett, Adam Trischler, Hal Daumé III, Kaheer Suleman, Alexandra Olteanu
NAACL 2022 | May 2022
-
Designing for Responsible Trust in AI Systems: A Communication Perspective
Q. Vera Liao, S. Shyam Sundar
FAccT 2022 | April 2022
Preprint
-
Responsible Language Technologies: Foreseeing and Mitigating Harms
Su Lin Blodgett, Q. Vera Liao, Alexandra Olteanu, Rada Mihalcea, Michael Muller, Morgan Klaus Scheuerman, Chenhao Tan, Qian Yang
CHI 2022 | April 2022
-
-
-
ADEPT: An Adjective-Dependent Plausibility Task
Ali Emami, Ian Porada, Alexandra Olteanu, Kaheer Suleman, Adam Trischler, Jackie Chi Kit Cheung
ACL-IJCNLP 2021 | August 2021
-
Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets
Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, Hanna Wallach
ACL-IJCNLP 2021 | August 2021
-
Responsible computing during COVID-19 and beyond
Solon Barocas, Asia J. Biega, Margarita Boyarskaya, Kate Crawford, Hal Daumé III, Miro Dudík, Benjamin Fish, Mary L. Gray, Brent Hecht, Alexandra Olteanu, Forough Poursabzi-Sangdeh, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach, Marion Zepf
Communications of the ACM | July 2021, Vol 67(7): pp. 30-32
https://doi.org/10.1145/3466612
-
Measuring algorithmically infused societies
Claudia Wagner, Markus Strohmaier, Alexandra Olteanu, Emre Kiciman, Noshir Contractor, Tina Eliassi-Rad
Nature | June 2021, Vol 595: pp. 197-204
-
«I Can’t Reply with That»: Characterizing Problematic Email Reply Suggestions
Ronald Robertson, Alexandra Olteanu, Fernando Diaz, Milad Shokouhi, Peter Bailey
CHI Conference on Human Factors in Computing Systems (CHI ’21) | May 2021
-
-
-
Overcoming Failures of Imagination in AI Infused System Development and Deployment
Margarita Boyarskaya, Alexandra Olteanu, Kate Crawford
In the Navigating the Broader Impacts of AI Research Workshop at NeurIPS 2020 | December 2020
-
When Are Search Completion Suggestions Problematic?
Alexandra Olteanu, Fernando Diaz, Gabriella Kazai
Computer Supported Collaborative Work and Social Computing (CSCW) | August 2020
CSCW 2020 Honorable Mention
CSCW 2020 Honorable Mention
-
Operationalizing the Legal Principle of Data Minimization for Personalization
Asia J. Biega, Peter Potash, Hal Daumé III, Fernando Diaz, Michèle Finck
SIGIR 2020 | July 2020
-
Evaluating Stochastic Rankings with Expected Exposure
Fernando Diaz, Bhaskar Mitra, Michael D. Ekstrand, Asia J. Biega, Ben Carterette
Proceedings of the 29th ACM International Conference on Information and Knowledge Management (CIKM) | July 2020
Best Long Research Paper Nominee
-
On the Social and Technical Challenges of Web Search Autosuggestion Moderation
T. J. Hazen, Alexandra Olteanu, Gabriella Kazai, Fernando Diaz, Michael Golebiewski
July 2020
-
External Information Sharing on Health Forums: An Exploration
Dana Nguyen, Alexandra Olteanu, Emre Kiciman
Intl. Conf. on Web and Social Media (ICWSM) | June 2020
-
Towards Query Logs For Privacy Studies: On Deriving Search Queries From Questions
Asia J. Biega, Jana Schmidt, Rishiraj Saha Roy
ECIR 2020 | April 2020
-
Co-Designing Checklists to Understand Organizational Challenges and Opportunities around Fairness in AI
Michael Madaio, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach
CHI Conference on Human Factors in Computing Systems | March 2020
CHI 2020 Best Paper Award
-
Privacy Norms and Preferences for Photos Posted Online
Roberto Hoyle, Luke Stark, Qatrunnada Ismail, David Crandall, Apu Kapadia, Denise Anthony
ACM Transactions on Computer-Human Interaction | March 2020
-
When not to design, build, or deploy
Solon Barocas, Asia J. Biega, Benjamin Fish, Jędrzej Niklas, Luke Stark
2020 Conference on Fairness, Accountability, and Transparency (FAT* ’20) | January 2020
-
-
-
FACTS-IR: Fairness, Accountability, Confidentiality, Transparency, and Safety in Information Retrieval
Alexandra Olteanu, Jean Garcia-Gathright, Maarten de Rijke, Michael D. Ekstrand
ACM SIGIR Forum | December 2019, Vol 53(2): pp. 20
-
Report on the First HIPstIR Workshop on the Future of Information Retrieval
Laura Dietz, Bhaskar Mitra, Jeremy Pickens, Hana Anber, Sandeep Avula, Asia J. Biega, Adrian Boteanu, Shubham Chatterjee, Jeff Dalton, Shiri Dori-Hacohen, John Foley, Henry Feild, Ben Gamari, Rosie Jones, Pallika Kanani, Sumanta Kashyapi, Widad Machmouchi, Matthew Mitsui, Steve Nole, Alexandre Tachard Passos, Jordan Ramsdell, Adam Roegiest, David Smith, Alessandro Sordoni
SIGIR Forum | December 2019, Vol 53(2): pp. 62-75
-
Overview of the TREC 2019 Fair Ranking Track
Asia J. Biega, Fernando Diaz, Michael D. Ekstrand, Sebastian Kohlmeier
TREC 2019 | November 2019
-
Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries
Alexandra Olteanu, Carlos Castillo, Fernando Diaz, Emre Kiciman
Frontiers in Big Data | July 2019, Vol 2
-
Gaps in Information Access in Social Networks
Benjamin Fish, Ashkan Bashardoust, danah boyd, Sorelle A. Friedler, Carlos Scheidegger, Suresh Venkatasubramanian
The Web Conference 2019 | May 2019
-
-
-
Rigor in AI: Doing Rigorous AI Work Requires a Broader, Responsible AI-Informed Conception of Rigor
Alexandra Olteanu, Su Lin Blodgett, Agathe Balayn, Angelina Wang, Fernando Diaz, Flavio du Pin Calmon, Margaret Mitchell, Michael Ekstrand, Reuben Binns, Solon Barocas
June 2025
-
Canvil: Designerly Adaptation for LLM-Powered User Experiences
K. Feng, Q. Vera Liao, Ziang Xiao, Jennifer Wortman Vaughan, Amy Zhang, David W. McDonald
CHI 2025 | April 2025
-
Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistencies
Sunnie S. Y. Kim, Jennifer Wortman Vaughan, Q. Vera Liao, Tania Lombrozo, Olga Russakovsky
CHI 2025 | April 2025
Honorable Mention, CHI 2025
-
A Taxonomy of Linguistic Expressions That Contribute To Anthropomorphism of Language Technologies
Alicia DeVrio, Myra Cheng, Lisa Egede, Alexandra Olteanu, Su Lin Blodgett
CHI 2025 | April 2025
-
«I Am the One and Only, Your Cyber BFF»: Understanding the Impact of GenAI Requires Understanding the Impact of Anthropomorphic AI
Myra Cheng, Alicia DeVrio, Lisa Egede, Su Lin Blodgett, Alexandra Olteanu
April 2025
-
Generation Probabilities Are Not Enough: Uncertainty Highlighting in AI Code Completions
Helena Vasconcelos, Gagan Bansal, Adam Fourney, Q. Vera Liao, Jennifer Wortman Vaughan
ToCHI | April 2025, Vol 32(1)
-
Position: Evaluating Generative AI Systems is a Social Science Measurement Challenge
Hanna Wallach, Meera Desai, A. Feder Cooper, Angelina Wang, Chad Atalla, Solon Barocas, Su Lin Blodgett, Alex Chouldechova, Emily Corvi, Alex Dow, Jean Garcia-Gathright, Alexandra Olteanu, Nick Pangakis, Stefanie Reed, Emily Sheng, Dan Vann, Jennifer Wortman Vaughan, Matthew Vogel, Hannah Washington, Abigail Z. Jacobs
ICML 2025 | January 2025
-
Gaps Between Research and Practice When Measuring Representational Harms Caused by LLM-Based Systems
Emma Harvey, Emily Sheng, Su Lin Blodgett, Alex Chouldechova, Jean Garcia-Gathright, Alexandra Olteanu, Hanna Wallach
November 2024
-
Evaluating Generative AI Systems is a Social Science Measurement Challenge
Hanna Wallach, Meera Desai, Nick Pangakis, A. Feder Cooper, Angelina Wang, Solon Barocas, Alex Chouldechova, Chad Atalla, Su Lin Blodgett, Emily Corvi, Alex Dow, Jean Garcia-Gathright, Alexandra Olteanu, Stefanie Reed, Emily Sheng, Dan Vann, Jennifer Wortman Vaughan, Matthew Vogel, Hannah Washington, Abigail Z. Jacobs
November 2024
-
«I’m Not Sure, But…»: Examining the Impact of Large Language Models’ Uncertainty Expression on User Reliance and Trust
Sunnie S. Y. Kim, Q. Vera Liao, Mihaela Vorvoreanu, Stephanie Ballard, Jennifer Wortman Vaughan
7th ACM Conference on Fairness, Accountability, and Transparency (FAccT 2024) | June 2024
-
AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap
Q. Vera Liao, Jennifer Wortman Vaughan
Harvard Data Science Review | February 2024
-
Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations
Valerie Chen, Q. Vera Liao, Jennifer Wortman Vaughan, Gagan Bansal
Proc. ACM Hum.-Comput. Interact. | October 2023, Vol 7
Article 370
-
FairPrism: Evaluating Fairness-Related Harms in Text Generation
Eve Fleisig, Aubrie Amstutz, Chad Atalla, Su Lin Blodgett, Hal Daumé III, Alexandra Olteanu, Emily Sheng, Dan Vann, Hanna Wallach
ACL 2023 | July 2023
-
AHA! Facilitating AI Impact Assessment by Generating Examples of Harms
Zana Buçinca, Chau Minh Pham, Maurice Jakesch, Marco Tulio Ribeiro, Alexandra Olteanu, Saleema Amershi
June 2023
-
Can Workers Meaningfully Consent To Workplace Wellbeing Technologies?
Shreya Chowdhary, Anna Kawakami, Mary L. Gray, Jina Suh, Alexandra Olteanu, Koustuv Saha
2023 ACM Conference on Fairness, Accountability, and Transparency | June 2023
-
The Kitmus Test: Evaluating Knowledge Integration From Multiple Sources In Natural Language Understanding Systems
Akshatha Arodi, Martin Pömsl, Kaheer Suleman, Adam Trischler, Alexandra Olteanu, Jackie Chi Kit Cheung
ACL 2023 | May 2023
-
Designerly Understanding: Information Needs for Model Transparency to Support Design Ideation for AI-Powered User Experience
Q. Vera Liao, Hariharan Subramonyam, Jennifer Wang, Jennifer Wortman Vaughan
2023 ACM CHI Conference on Human Factors in Computing Systems (CHI 2023) | April 2023
-
Taxonomizing and Measuring Representational Harms: A Look at Image Tagging
Jared Katzman, Angelina Wang, Morgan Scheuerman, Su Lin Blodgett, Kristen Laird, Hanna Wallach, Solon Barocas
AAAI 2022 | February 2023
-
Generation Probabilities are Not Enough: Improving Error Highlighting for AI Code Suggestions
Helena Vasconcelos, Gagan Bansal, Adam Fourney, Q. Vera Liao, Jennifer Wortman Vaughan
NeurIPS Workshop on Human-Centered AI | October 2022
-
Connecting Algorithmic Research and Usage Contexts: A Perspective of Contextualized Evaluation for Explainable AI
Q. Vera Liao, Yunfeng Zhang, Ronny Luss, Finale Doshi-Velez, Amit Dhurandhar
AAAI Conference on Human Computation and Crowdsourcing | October 2022
Best Paper Award, Honorable Mention
-
Examining Responsibility and Deliberation in AI Impact Statements and Ethics Reviews
David Liu, Priyanka Nanayakkara, Sarah Ariyan Sakha, Grace Abuhamad, Su Lin Blodgett, Nicholas Diakopoulos, Jessica R. Hullman, Tina Eliassi-Rad
AIES 2022 | July 2022
-
How Different Groups Prioritize Ethical Values for Responsible AI
Maurice Jakesch, Zana Buçinca, Saleema Amershi, Alexandra Olteanu
FAccT 2022 | May 2022
-
Designing for Responsible Trust in AI Systems: A Communication Perspective
Q. Vera Liao, S. Shyam Sundar
FAccT 2022 | April 2022
Preprint
-
ADEPT: An Adjective-Dependent Plausibility Task
Ali Emami, Ian Porada, Alexandra Olteanu, Kaheer Suleman, Adam Trischler, Jackie Chi Kit Cheung
ACL-IJCNLP 2021 | August 2021
-
Responsible computing during COVID-19 and beyond
Solon Barocas, Asia J. Biega, Margarita Boyarskaya, Kate Crawford, Hal Daumé III, Miro Dudík, Benjamin Fish, Mary L. Gray, Brent Hecht, Alexandra Olteanu, Forough Poursabzi-Sangdeh, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach, Marion Zepf
Communications of the ACM | July 2021, Vol 67(7): pp. 30-32
https://doi.org/10.1145/3466612
-
Overcoming Failures of Imagination in AI Infused System Development and Deployment
Margarita Boyarskaya, Alexandra Olteanu, Kate Crawford
In the Navigating the Broader Impacts of AI Research Workshop at NeurIPS 2020 | December 2020
-
Operationalizing the Legal Principle of Data Minimization for Personalization
Asia J. Biega, Peter Potash, Hal Daumé III, Fernando Diaz, Michèle Finck
SIGIR 2020 | July 2020
-
Evaluating Stochastic Rankings with Expected Exposure
Fernando Diaz, Bhaskar Mitra, Michael D. Ekstrand, Asia J. Biega, Ben Carterette
Proceedings of the 29th ACM International Conference on Information and Knowledge Management (CIKM) | July 2020
Best Long Research Paper Nominee
-
External Information Sharing on Health Forums: An Exploration
Dana Nguyen, Alexandra Olteanu, Emre Kiciman
Intl. Conf. on Web and Social Media (ICWSM) | June 2020
-
Towards Query Logs For Privacy Studies: On Deriving Search Queries From Questions
Asia J. Biega, Jana Schmidt, Rishiraj Saha Roy
ECIR 2020 | April 2020
-
Co-Designing Checklists to Understand Organizational Challenges and Opportunities around Fairness in AI
Michael Madaio, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach
CHI Conference on Human Factors in Computing Systems | March 2020
CHI 2020 Best Paper Award
-
When not to design, build, or deploy
Solon Barocas, Asia J. Biega, Benjamin Fish, Jędrzej Niklas, Luke Stark
2020 Conference on Fairness, Accountability, and Transparency (FAT* ’20) | January 2020
-
FACTS-IR: Fairness, Accountability, Confidentiality, Transparency, and Safety in Information Retrieval
Alexandra Olteanu, Jean Garcia-Gathright, Maarten de Rijke, Michael D. Ekstrand
ACM SIGIR Forum | December 2019, Vol 53(2): pp. 20
-
Report on the First HIPstIR Workshop on the Future of Information Retrieval
Laura Dietz, Bhaskar Mitra, Jeremy Pickens, Hana Anber, Sandeep Avula, Asia J. Biega, Adrian Boteanu, Shubham Chatterjee, Jeff Dalton, Shiri Dori-Hacohen, John Foley, Henry Feild, Ben Gamari, Rosie Jones, Pallika Kanani, Sumanta Kashyapi, Widad Machmouchi, Matthew Mitsui, Steve Nole, Alexandre Tachard Passos, Jordan Ramsdell, Adam Roegiest, David Smith, Alessandro Sordoni
SIGIR Forum | December 2019, Vol 53(2): pp. 62-75
-
Overview of the TREC 2019 Fair Ranking Track
Asia J. Biega, Fernando Diaz, Michael D. Ekstrand, Sebastian Kohlmeier
TREC 2019 | November 2019
-
Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries
Alexandra Olteanu, Carlos Castillo, Fernando Diaz, Emre Kiciman
Frontiers in Big Data | July 2019, Vol 2
-
Gaps in Information Access in Social Networks
Benjamin Fish, Ashkan Bashardoust, danah boyd, Sorelle A. Friedler, Carlos Scheidegger, Suresh Venkatasubramanian
The Web Conference 2019 | May 2019
-
-
-
Understanding and Meeting Practitioner Needs When Measuring Representational Harms Caused by LLM-Based Systems
Emma Harvey, Emily Sheng, Su Lin Blodgett, Alex Chouldechova, Jean Garcia-Gathright, Alexandra Olteanu, Hanna Wallach
Findings of the Association for Computational Linguistics: ACL 2025 | July 2025
-
Dehumanizing Machines: Mitigating Anthropomorphic Behaviors in Text Generation Systems
Myra Cheng, Su Lin Blodgett, Alicia DeVrio, Lisa Egede, Alexandra Olteanu
2025 Meeting of the Association for Computational Linguistics | July 2025
SAC Highlight
-
Measuring Machine Learning Harms from Stereotypes: Requires Understanding Who is Being Harmed by Which Errors in What Ways
Angelina Wang, Xuechunzi Bai, Solon Barocas, Su Lin Blodgett
2025 ACM Conference on Fairness, Accountability, and Transparency | June 2025
-
«I Am the One and Only, Your Cyber BFF»: Understanding the Impact of GenAI Requires Understanding the Impact of Anthropomorphic AI
Myra Cheng, Alicia DeVrio, Lisa Egede, Su Lin Blodgett, Alexandra Olteanu
April 2025
-
AI Automatons: AI Systems Intended to Imitate Humans
Alexandra Olteanu, Solon Barocas, Su Lin Blodgett, Lisa Egede, Alicia DeVrio, Myra Cheng
March 2025
-
Metrics for What, Metrics for Whom: Assessing Actionability of Bias Evaluation Metrics in NLP
Pieter Delobelle, Giuseppe Attanasio, Debora Nozza, Su Lin Blodgett, Zeerak Talat
2024 Empirical Methods in Natural Language Processing | November 2024
-
The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels
Eve Fleisig, Su Lin Blodgett, Dan Klein, Zeerak Talat
NAACL 2024 | June 2024
-
“One-Size-Fits-All”? Examining Expectations around What Constitute “Fair” or “Good” NLG System Behaviors
Li Lucy, Su Lin Blodgett, Milad Shokouhi, Hanna Wallach, Alexandra Olteanu
NAACL 2024 | June 2024
-
Responsible AI Considerations in Text Summarization Research: A Review of Current Practices
Yu Lu Liu, Meng Cao, Su Lin Blodgett, Jackie Chi Kit Cheung, Alexandra Olteanu, Adam Trischler
Findings of EMNLP 2023 | December 2023
-
This Prompt is Measuring : Evaluating Bias Evaluation in Language Models
Seraphina Goldfarb-Tarrant, Eddie Ungless, Esma Balkir, Su Lin Blodgett
Findings of the Association for Computational Linguistics: ACL 2023 | July 2023
-
Deconstructing NLG Evaluation: Evaluation Practices, Assumptions, and Their Implications
Kaitlyn Zhou, Su Lin Blodgett, Adam Trischler, Hal Daumé III, Kaheer Suleman, Alexandra Olteanu
NAACL 2022 | May 2022
-
Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets
Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, Hanna Wallach
ACL-IJCNLP 2021 | August 2021
-
«I Can’t Reply with That»: Characterizing Problematic Email Reply Suggestions
Ronald Robertson, Alexandra Olteanu, Fernando Diaz, Milad Shokouhi, Peter Bailey
CHI Conference on Human Factors in Computing Systems (CHI ’21) | May 2021
-
When Are Search Completion Suggestions Problematic?
Alexandra Olteanu, Fernando Diaz, Gabriella Kazai
Computer Supported Collaborative Work and Social Computing (CSCW) | August 2020
CSCW 2020 Honorable Mention
CSCW 2020 Honorable Mention
-
On the Social and Technical Challenges of Web Search Autosuggestion Moderation
T. J. Hazen, Alexandra Olteanu, Gabriella Kazai, Fernando Diaz, Michael Golebiewski
July 2020
-
Towards Query Logs For Privacy Studies: On Deriving Search Queries From Questions
Asia J. Biega, Jana Schmidt, Rishiraj Saha Roy
ECIR 2020 | April 2020
-
Overview of the TREC 2019 Fair Ranking Track
Asia J. Biega, Fernando Diaz, Michael D. Ekstrand, Sebastian Kohlmeier
TREC 2019 | November 2019
-
-
-
Understanding and Meeting Practitioner Needs When Measuring Representational Harms Caused by LLM-Based Systems
Emma Harvey, Emily Sheng, Su Lin Blodgett, Alex Chouldechova, Jean Garcia-Gathright, Alexandra Olteanu, Hanna Wallach
Findings of the Association for Computational Linguistics: ACL 2025 | July 2025
-
Dehumanizing Machines: Mitigating Anthropomorphic Behaviors in Text Generation Systems
Myra Cheng, Su Lin Blodgett, Alicia DeVrio, Lisa Egede, Alexandra Olteanu
2025 Meeting of the Association for Computational Linguistics | July 2025
SAC Highlight
-
Measuring Machine Learning Harms from Stereotypes: Requires Understanding Who is Being Harmed by Which Errors in What Ways
Angelina Wang, Xuechunzi Bai, Solon Barocas, Su Lin Blodgett
2025 ACM Conference on Fairness, Accountability, and Transparency | June 2025
-
Rigor in AI: Doing Rigorous AI Work Requires a Broader, Responsible AI-Informed Conception of Rigor
Alexandra Olteanu, Su Lin Blodgett, Agathe Balayn, Angelina Wang, Fernando Diaz, Flavio du Pin Calmon, Margaret Mitchell, Michael Ekstrand, Reuben Binns, Solon Barocas
June 2025
-
«I Am the One and Only, Your Cyber BFF»: Understanding the Impact of GenAI Requires Understanding the Impact of Anthropomorphic AI
Myra Cheng, Alicia DeVrio, Lisa Egede, Su Lin Blodgett, Alexandra Olteanu
April 2025
-
AI Automatons: AI Systems Intended to Imitate Humans
Alexandra Olteanu, Solon Barocas, Su Lin Blodgett, Lisa Egede, Alicia DeVrio, Myra Cheng
March 2025
-
Position: Evaluating Generative AI Systems is a Social Science Measurement Challenge
Hanna Wallach, Meera Desai, A. Feder Cooper, Angelina Wang, Chad Atalla, Solon Barocas, Su Lin Blodgett, Alex Chouldechova, Emily Corvi, Alex Dow, Jean Garcia-Gathright, Alexandra Olteanu, Nick Pangakis, Stefanie Reed, Emily Sheng, Dan Vann, Jennifer Wortman Vaughan, Matthew Vogel, Hannah Washington, Abigail Z. Jacobs
ICML 2025 | January 2025
-
Gaps Between Research and Practice When Measuring Representational Harms Caused by LLM-Based Systems
Emma Harvey, Emily Sheng, Su Lin Blodgett, Alex Chouldechova, Jean Garcia-Gathright, Alexandra Olteanu, Hanna Wallach
November 2024
-
Evaluating Generative AI Systems is a Social Science Measurement Challenge
Hanna Wallach, Meera Desai, Nick Pangakis, A. Feder Cooper, Angelina Wang, Solon Barocas, Alex Chouldechova, Chad Atalla, Su Lin Blodgett, Emily Corvi, Alex Dow, Jean Garcia-Gathright, Alexandra Olteanu, Stefanie Reed, Emily Sheng, Dan Vann, Jennifer Wortman Vaughan, Matthew Vogel, Hannah Washington, Abigail Z. Jacobs
November 2024
-
Can Workers Meaningfully Consent To Workplace Wellbeing Technologies?
Shreya Chowdhary, Anna Kawakami, Mary L. Gray, Jina Suh, Alexandra Olteanu, Koustuv Saha
2023 ACM Conference on Fairness, Accountability, and Transparency | June 2023
-
Responsible computing during COVID-19 and beyond
Solon Barocas, Asia J. Biega, Margarita Boyarskaya, Kate Crawford, Hal Daumé III, Miro Dudík, Benjamin Fish, Mary L. Gray, Brent Hecht, Alexandra Olteanu, Forough Poursabzi-Sangdeh, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach, Marion Zepf
Communications of the ACM | July 2021, Vol 67(7): pp. 30-32
https://doi.org/10.1145/3466612
-
Measuring algorithmically infused societies
Claudia Wagner, Markus Strohmaier, Alexandra Olteanu, Emre Kiciman, Noshir Contractor, Tina Eliassi-Rad
Nature | June 2021, Vol 595: pp. 197-204
-
Overcoming Failures of Imagination in AI Infused System Development and Deployment
Margarita Boyarskaya, Alexandra Olteanu, Kate Crawford
In the Navigating the Broader Impacts of AI Research Workshop at NeurIPS 2020 | December 2020
-
Operationalizing the Legal Principle of Data Minimization for Personalization
Asia J. Biega, Peter Potash, Hal Daumé III, Fernando Diaz, Michèle Finck
SIGIR 2020 | July 2020
-
When not to design, build, or deploy
Solon Barocas, Asia J. Biega, Benjamin Fish, Jędrzej Niklas, Luke Stark
2020 Conference on Fairness, Accountability, and Transparency (FAT* ’20) | January 2020
-
Overview of the TREC 2019 Fair Ranking Track
Asia J. Biega, Fernando Diaz, Michael D. Ekstrand, Sebastian Kohlmeier
TREC 2019 | November 2019
-
-
-
Canvil: Designerly Adaptation for LLM-Powered User Experiences
K. Feng, Q. Vera Liao, Ziang Xiao, Jennifer Wortman Vaughan, Amy Zhang, David W. McDonald
CHI 2025 | April 2025
-
Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistencies
Sunnie S. Y. Kim, Jennifer Wortman Vaughan, Q. Vera Liao, Tania Lombrozo, Olga Russakovsky
CHI 2025 | April 2025
Honorable Mention, CHI 2025
-
A Taxonomy of Linguistic Expressions That Contribute To Anthropomorphism of Language Technologies
Alicia DeVrio, Myra Cheng, Lisa Egede, Alexandra Olteanu, Su Lin Blodgett
CHI 2025 | April 2025
-
Generation Probabilities Are Not Enough: Uncertainty Highlighting in AI Code Completions
Helena Vasconcelos, Gagan Bansal, Adam Fourney, Q. Vera Liao, Jennifer Wortman Vaughan
ToCHI | April 2025, Vol 32(1)
-
AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap
Q. Vera Liao, Jennifer Wortman Vaughan
Harvard Data Science Review | February 2024
-
Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations
Valerie Chen, Q. Vera Liao, Jennifer Wortman Vaughan, Gagan Bansal
Proc. ACM Hum.-Comput. Interact. | October 2023, Vol 7
Article 370
-
AHA! Facilitating AI Impact Assessment by Generating Examples of Harms
Zana Buçinca, Chau Minh Pham, Maurice Jakesch, Marco Tulio Ribeiro, Alexandra Olteanu, Saleema Amershi
June 2023
-
Can Workers Meaningfully Consent To Workplace Wellbeing Technologies?
Shreya Chowdhary, Anna Kawakami, Mary L. Gray, Jina Suh, Alexandra Olteanu, Koustuv Saha
2023 ACM Conference on Fairness, Accountability, and Transparency | June 2023
-
The Kitmus Test: Evaluating Knowledge Integration From Multiple Sources In Natural Language Understanding Systems
Akshatha Arodi, Martin Pömsl, Kaheer Suleman, Adam Trischler, Alexandra Olteanu, Jackie Chi Kit Cheung
ACL 2023 | May 2023
-
fAIlureNotes: Supporting Designers in Understanding the Limits of AI Models for Computer Vision Tasks
Steven Moore, Q. Vera Liao, Hariharan Subramonyam
CHI 2023 | April 2023
-
Designerly Understanding: Information Needs for Model Transparency to Support Design Ideation for AI-Powered User Experience
Q. Vera Liao, Hariharan Subramonyam, Jennifer Wang, Jennifer Wortman Vaughan
2023 ACM CHI Conference on Human Factors in Computing Systems (CHI 2023) | April 2023
-
Sensing Wellbeing in the Workplace, Why and For Whom? Envisioning Impacts with Organizational Stakeholders
Anna Kawakami, Shreya Chowdhary, Shamsi T. Iqbal, Q. Liao, Alexandra Olteanu, J. Suh, Koustuv Saha, Alexandra Olteanu, Jina Suh
Proceedings of the ACM on Human-Computer Interaction | March 2023, Vol 7: pp. 1-33
-
Powering an AI Chatbot with Expert Sourcing to Support Credible Information Access
Ziang Xiao, Q. Vera Liao, Michelle X. Zhou, Tyrone Grandison, Yunyao Li
IUI 2023 | March 2023
-
Generation Probabilities are Not Enough: Improving Error Highlighting for AI Code Suggestions
Helena Vasconcelos, Gagan Bansal, Adam Fourney, Q. Vera Liao, Jennifer Wortman Vaughan
NeurIPS Workshop on Human-Centered AI | October 2022
-
Connecting Algorithmic Research and Usage Contexts: A Perspective of Contextualized Evaluation for Explainable AI
Q. Vera Liao, Yunfeng Zhang, Ronny Luss, Finale Doshi-Velez, Amit Dhurandhar
AAAI Conference on Human Computation and Crowdsourcing | October 2022
Best Paper Award, Honorable Mention
-
How Different Groups Prioritize Ethical Values for Responsible AI
Maurice Jakesch, Zana Buçinca, Saleema Amershi, Alexandra Olteanu
FAccT 2022 | May 2022
-
Designing for Responsible Trust in AI Systems: A Communication Perspective
Q. Vera Liao, S. Shyam Sundar
FAccT 2022 | April 2022
Preprint
-
Responsible Language Technologies: Foreseeing and Mitigating Harms
Su Lin Blodgett, Q. Vera Liao, Alexandra Olteanu, Rada Mihalcea, Michael Muller, Morgan Klaus Scheuerman, Chenhao Tan, Qian Yang
CHI 2022 | April 2022
-
Responsible computing during COVID-19 and beyond
Solon Barocas, Asia J. Biega, Margarita Boyarskaya, Kate Crawford, Hal Daumé III, Miro Dudík, Benjamin Fish, Mary L. Gray, Brent Hecht, Alexandra Olteanu, Forough Poursabzi-Sangdeh, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach, Marion Zepf
Communications of the ACM | July 2021, Vol 67(7): pp. 30-32
https://doi.org/10.1145/3466612
-
Measuring algorithmically infused societies
Claudia Wagner, Markus Strohmaier, Alexandra Olteanu, Emre Kiciman, Noshir Contractor, Tina Eliassi-Rad
Nature | June 2021, Vol 595: pp. 197-204
-
«I Can’t Reply with That»: Characterizing Problematic Email Reply Suggestions
Ronald Robertson, Alexandra Olteanu, Fernando Diaz, Milad Shokouhi, Peter Bailey
CHI Conference on Human Factors in Computing Systems (CHI ’21) | May 2021
-
When Are Search Completion Suggestions Problematic?
Alexandra Olteanu, Fernando Diaz, Gabriella Kazai
Computer Supported Collaborative Work and Social Computing (CSCW) | August 2020
CSCW 2020 Honorable Mention
CSCW 2020 Honorable Mention
-
Co-Designing Checklists to Understand Organizational Challenges and Opportunities around Fairness in AI
Michael Madaio, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach
CHI Conference on Human Factors in Computing Systems | March 2020
CHI 2020 Best Paper Award
-
Privacy Norms and Preferences for Photos Posted Online
Roberto Hoyle, Luke Stark, Qatrunnada Ismail, David Crandall, Apu Kapadia, Denise Anthony
ACM Transactions on Computer-Human Interaction | March 2020
-
When not to design, build, or deploy
Solon Barocas, Asia J. Biega, Benjamin Fish, Jędrzej Niklas, Luke Stark
2020 Conference on Fairness, Accountability, and Transparency (FAT* ’20) | January 2020
-
-
-
Overcoming Failures of Imagination in AI Infused System Development and Deployment
Margarita Boyarskaya, Alexandra Olteanu, Kate Crawford
In the Navigating the Broader Impacts of AI Research Workshop at NeurIPS 2020 | December 2020
-
-
-
When Are Search Completion Suggestions Problematic?
Alexandra Olteanu, Fernando Diaz, Gabriella Kazai
Computer Supported Collaborative Work and Social Computing (CSCW) | August 2020
CSCW 2020 Honorable Mention
CSCW 2020 Honorable Mention
-
Operationalizing the Legal Principle of Data Minimization for Personalization
Asia J. Biega, Peter Potash, Hal Daumé III, Fernando Diaz, Michèle Finck
SIGIR 2020 | July 2020
-
On the Social and Technical Challenges of Web Search Autosuggestion Moderation
T. J. Hazen, Alexandra Olteanu, Gabriella Kazai, Fernando Diaz, Michael Golebiewski
July 2020
-
Evaluating Stochastic Rankings with Expected Exposure
Fernando Diaz, Bhaskar Mitra, Michael D. Ekstrand, Asia J. Biega, Ben Carterette
Proceedings of the 29th ACM International Conference on Information and Knowledge Management (CIKM) | July 2020
Best Long Research Paper Nominee
-
Towards Query Logs For Privacy Studies: On Deriving Search Queries From Questions
Asia J. Biega, Jana Schmidt, Rishiraj Saha Roy
ECIR 2020 | April 2020
-
FACTS-IR: Fairness, Accountability, Confidentiality, Transparency, and Safety in Information Retrieval
Alexandra Olteanu, Jean Garcia-Gathright, Maarten de Rijke, Michael D. Ekstrand
ACM SIGIR Forum | December 2019, Vol 53(2): pp. 20
-
Report on the First HIPstIR Workshop on the Future of Information Retrieval
Laura Dietz, Bhaskar Mitra, Jeremy Pickens, Hana Anber, Sandeep Avula, Asia J. Biega, Adrian Boteanu, Shubham Chatterjee, Jeff Dalton, Shiri Dori-Hacohen, John Foley, Henry Feild, Ben Gamari, Rosie Jones, Pallika Kanani, Sumanta Kashyapi, Widad Machmouchi, Matthew Mitsui, Steve Nole, Alexandre Tachard Passos, Jordan Ramsdell, Adam Roegiest, David Smith, Alessandro Sordoni
SIGIR Forum | December 2019, Vol 53(2): pp. 62-75
-
Overview of the TREC 2019 Fair Ranking Track
Asia J. Biega, Fernando Diaz, Michael D. Ekstrand, Sebastian Kohlmeier
TREC 2019 | November 2019
-
-
-
Operationalizing the Legal Principle of Data Minimization for Personalization
Asia J. Biega, Peter Potash, Hal Daumé III, Fernando Diaz, Michèle Finck
SIGIR 2020 | July 2020
-
Towards Query Logs For Privacy Studies: On Deriving Search Queries From Questions
Asia J. Biega, Jana Schmidt, Rishiraj Saha Roy
ECIR 2020 | April 2020
-
Privacy Norms and Preferences for Photos Posted Online
Roberto Hoyle, Luke Stark, Qatrunnada Ismail, David Crandall, Apu Kapadia, Denise Anthony
ACM Transactions on Computer-Human Interaction | March 2020
-
-
-
«I Am the One and Only, Your Cyber BFF»: Understanding the Impact of GenAI Requires Understanding the Impact of Anthropomorphic AI
Myra Cheng, Alicia DeVrio, Lisa Egede, Su Lin Blodgett, Alexandra Olteanu
April 2025
-
Gaps Between Research and Practice When Measuring Representational Harms Caused by LLM-Based Systems
Emma Harvey, Emily Sheng, Su Lin Blodgett, Alex Chouldechova, Jean Garcia-Gathright, Alexandra Olteanu, Hanna Wallach
November 2024
-
Evaluating Generative AI Systems is a Social Science Measurement Challenge
Hanna Wallach, Meera Desai, Nick Pangakis, A. Feder Cooper, Angelina Wang, Solon Barocas, Alex Chouldechova, Chad Atalla, Su Lin Blodgett, Emily Corvi, Alex Dow, Jean Garcia-Gathright, Alexandra Olteanu, Stefanie Reed, Emily Sheng, Dan Vann, Jennifer Wortman Vaughan, Matthew Vogel, Hannah Washington, Abigail Z. Jacobs
November 2024
-
AHA! Facilitating AI Impact Assessment by Generating Examples of Harms
Zana Buçinca, Chau Minh Pham, Maurice Jakesch, Marco Tulio Ribeiro, Alexandra Olteanu, Saleema Amershi
June 2023
-
-
-
Understanding and Meeting Practitioner Needs When Measuring Representational Harms Caused by LLM-Based Systems
Emma Harvey, Emily Sheng, Su Lin Blodgett, Alex Chouldechova, Jean Garcia-Gathright, Alexandra Olteanu, Hanna Wallach
Findings of the Association for Computational Linguistics: ACL 2025 | July 2025
-
Dehumanizing Machines: Mitigating Anthropomorphic Behaviors in Text Generation Systems
Myra Cheng, Su Lin Blodgett, Alicia DeVrio, Lisa Egede, Alexandra Olteanu
2025 Meeting of the Association for Computational Linguistics | July 2025
SAC Highlight
-
Measuring Machine Learning Harms from Stereotypes: Requires Understanding Who is Being Harmed by Which Errors in What Ways
Angelina Wang, Xuechunzi Bai, Solon Barocas, Su Lin Blodgett
2025 ACM Conference on Fairness, Accountability, and Transparency | June 2025
-
Canvil: Designerly Adaptation for LLM-Powered User Experiences
K. Feng, Q. Vera Liao, Ziang Xiao, Jennifer Wortman Vaughan, Amy Zhang, David W. McDonald
CHI 2025 | April 2025
-
Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistencies
Sunnie S. Y. Kim, Jennifer Wortman Vaughan, Q. Vera Liao, Tania Lombrozo, Olga Russakovsky
CHI 2025 | April 2025
Honorable Mention, CHI 2025
-
A Taxonomy of Linguistic Expressions That Contribute To Anthropomorphism of Language Technologies
Alicia DeVrio, Myra Cheng, Lisa Egede, Alexandra Olteanu, Su Lin Blodgett
CHI 2025 | April 2025
-
Position: Evaluating Generative AI Systems is a Social Science Measurement Challenge
Hanna Wallach, Meera Desai, A. Feder Cooper, Angelina Wang, Chad Atalla, Solon Barocas, Su Lin Blodgett, Alex Chouldechova, Emily Corvi, Alex Dow, Jean Garcia-Gathright, Alexandra Olteanu, Nick Pangakis, Stefanie Reed, Emily Sheng, Dan Vann, Jennifer Wortman Vaughan, Matthew Vogel, Hannah Washington, Abigail Z. Jacobs
ICML 2025 | January 2025
-
Metrics for What, Metrics for Whom: Assessing Actionability of Bias Evaluation Metrics in NLP
Pieter Delobelle, Giuseppe Attanasio, Debora Nozza, Su Lin Blodgett, Zeerak Talat
2024 Empirical Methods in Natural Language Processing | November 2024
-
The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels
Eve Fleisig, Su Lin Blodgett, Dan Klein, Zeerak Talat
NAACL 2024 | June 2024
-
“One-Size-Fits-All”? Examining Expectations around What Constitute “Fair” or “Good” NLG System Behaviors
Li Lucy, Su Lin Blodgett, Milad Shokouhi, Hanna Wallach, Alexandra Olteanu
NAACL 2024 | June 2024
-
«I’m Not Sure, But…»: Examining the Impact of Large Language Models’ Uncertainty Expression on User Reliance and Trust
Sunnie S. Y. Kim, Q. Vera Liao, Mihaela Vorvoreanu, Stephanie Ballard, Jennifer Wortman Vaughan
7th ACM Conference on Fairness, Accountability, and Transparency (FAccT 2024) | June 2024
-
Responsible AI Considerations in Text Summarization Research: A Review of Current Practices
Yu Lu Liu, Meng Cao, Su Lin Blodgett, Jackie Chi Kit Cheung, Alexandra Olteanu, Adam Trischler
Findings of EMNLP 2023 | December 2023
-
FairPrism: Evaluating Fairness-Related Harms in Text Generation
Eve Fleisig, Aubrie Amstutz, Chad Atalla, Su Lin Blodgett, Hal Daumé III, Alexandra Olteanu, Emily Sheng, Dan Vann, Hanna Wallach
ACL 2023 | July 2023
-
This Prompt is Measuring : Evaluating Bias Evaluation in Language Models
Seraphina Goldfarb-Tarrant, Eddie Ungless, Esma Balkir, Su Lin Blodgett
Findings of the Association for Computational Linguistics: ACL 2023 | July 2023
-
Can Workers Meaningfully Consent To Workplace Wellbeing Technologies?
Shreya Chowdhary, Anna Kawakami, Mary L. Gray, Jina Suh, Alexandra Olteanu, Koustuv Saha
2023 ACM Conference on Fairness, Accountability, and Transparency | June 2023
-
The Kitmus Test: Evaluating Knowledge Integration From Multiple Sources In Natural Language Understanding Systems
Akshatha Arodi, Martin Pömsl, Kaheer Suleman, Adam Trischler, Alexandra Olteanu, Jackie Chi Kit Cheung
ACL 2023 | May 2023
-
fAIlureNotes: Supporting Designers in Understanding the Limits of AI Models for Computer Vision Tasks
Steven Moore, Q. Vera Liao, Hariharan Subramonyam
CHI 2023 | April 2023
-
Designerly Understanding: Information Needs for Model Transparency to Support Design Ideation for AI-Powered User Experience
Q. Vera Liao, Hariharan Subramonyam, Jennifer Wang, Jennifer Wortman Vaughan
2023 ACM CHI Conference on Human Factors in Computing Systems (CHI 2023) | April 2023
-
Powering an AI Chatbot with Expert Sourcing to Support Credible Information Access
Ziang Xiao, Q. Vera Liao, Michelle X. Zhou, Tyrone Grandison, Yunyao Li
IUI 2023 | March 2023
-
Taxonomizing and Measuring Representational Harms: A Look at Image Tagging
Jared Katzman, Angelina Wang, Morgan Scheuerman, Su Lin Blodgett, Kristen Laird, Hanna Wallach, Solon Barocas
AAAI 2022 | February 2023
-
Generation Probabilities are Not Enough: Improving Error Highlighting for AI Code Suggestions
Helena Vasconcelos, Gagan Bansal, Adam Fourney, Q. Vera Liao, Jennifer Wortman Vaughan
NeurIPS Workshop on Human-Centered AI | October 2022
-
Connecting Algorithmic Research and Usage Contexts: A Perspective of Contextualized Evaluation for Explainable AI
Q. Vera Liao, Yunfeng Zhang, Ronny Luss, Finale Doshi-Velez, Amit Dhurandhar
AAAI Conference on Human Computation and Crowdsourcing | October 2022
Best Paper Award, Honorable Mention
-
Examining Responsibility and Deliberation in AI Impact Statements and Ethics Reviews
David Liu, Priyanka Nanayakkara, Sarah Ariyan Sakha, Grace Abuhamad, Su Lin Blodgett, Nicholas Diakopoulos, Jessica R. Hullman, Tina Eliassi-Rad
AIES 2022 | July 2022
-
How Different Groups Prioritize Ethical Values for Responsible AI
Maurice Jakesch, Zana Buçinca, Saleema Amershi, Alexandra Olteanu
FAccT 2022 | May 2022
-
Deconstructing NLG Evaluation: Evaluation Practices, Assumptions, and Their Implications
Kaitlyn Zhou, Su Lin Blodgett, Adam Trischler, Hal Daumé III, Kaheer Suleman, Alexandra Olteanu
NAACL 2022 | May 2022
-
Designing for Responsible Trust in AI Systems: A Communication Perspective
Q. Vera Liao, S. Shyam Sundar
FAccT 2022 | April 2022
Preprint
-
Responsible Language Technologies: Foreseeing and Mitigating Harms
Su Lin Blodgett, Q. Vera Liao, Alexandra Olteanu, Rada Mihalcea, Michael Muller, Morgan Klaus Scheuerman, Chenhao Tan, Qian Yang
CHI 2022 | April 2022
-
ADEPT: An Adjective-Dependent Plausibility Task
Ali Emami, Ian Porada, Alexandra Olteanu, Kaheer Suleman, Adam Trischler, Jackie Chi Kit Cheung
ACL-IJCNLP 2021 | August 2021
-
Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets
Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, Hanna Wallach
ACL-IJCNLP 2021 | August 2021
-
«I Can’t Reply with That»: Characterizing Problematic Email Reply Suggestions
Ronald Robertson, Alexandra Olteanu, Fernando Diaz, Milad Shokouhi, Peter Bailey
CHI Conference on Human Factors in Computing Systems (CHI ’21) | May 2021
-
Overcoming Failures of Imagination in AI Infused System Development and Deployment
Margarita Boyarskaya, Alexandra Olteanu, Kate Crawford
In the Navigating the Broader Impacts of AI Research Workshop at NeurIPS 2020 | December 2020
-
When Are Search Completion Suggestions Problematic?
Alexandra Olteanu, Fernando Diaz, Gabriella Kazai
Computer Supported Collaborative Work and Social Computing (CSCW) | August 2020
CSCW 2020 Honorable Mention
CSCW 2020 Honorable Mention
-
Operationalizing the Legal Principle of Data Minimization for Personalization
Asia J. Biega, Peter Potash, Hal Daumé III, Fernando Diaz, Michèle Finck
SIGIR 2020 | July 2020
-
Evaluating Stochastic Rankings with Expected Exposure
Fernando Diaz, Bhaskar Mitra, Michael D. Ekstrand, Asia J. Biega, Ben Carterette
Proceedings of the 29th ACM International Conference on Information and Knowledge Management (CIKM) | July 2020
Best Long Research Paper Nominee
-
External Information Sharing on Health Forums: An Exploration
Dana Nguyen, Alexandra Olteanu, Emre Kiciman
Intl. Conf. on Web and Social Media (ICWSM) | June 2020
-
Towards Query Logs For Privacy Studies: On Deriving Search Queries From Questions
Asia J. Biega, Jana Schmidt, Rishiraj Saha Roy
ECIR 2020 | April 2020
-
Co-Designing Checklists to Understand Organizational Challenges and Opportunities around Fairness in AI
Michael Madaio, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach
CHI Conference on Human Factors in Computing Systems | March 2020
CHI 2020 Best Paper Award
-
When not to design, build, or deploy
Solon Barocas, Asia J. Biega, Benjamin Fish, Jędrzej Niklas, Luke Stark
2020 Conference on Fairness, Accountability, and Transparency (FAT* ’20) | January 2020
-
Overview of the TREC 2019 Fair Ranking Track
Asia J. Biega, Fernando Diaz, Michael D. Ekstrand, Sebastian Kohlmeier
TREC 2019 | November 2019
-
Gaps in Information Access in Social Networks
Benjamin Fish, Ashkan Bashardoust, danah boyd, Sorelle A. Friedler, Carlos Scheidegger, Suresh Venkatasubramanian
The Web Conference 2019 | May 2019
-
-
-
Rigor in AI: Doing Rigorous AI Work Requires a Broader, Responsible AI-Informed Conception of Rigor
Alexandra Olteanu, Su Lin Blodgett, Agathe Balayn, Angelina Wang, Fernando Diaz, Flavio du Pin Calmon, Margaret Mitchell, Michael Ekstrand, Reuben Binns, Solon Barocas
June 2025
-
AI Automatons: AI Systems Intended to Imitate Humans
Alexandra Olteanu, Solon Barocas, Su Lin Blodgett, Lisa Egede, Alicia DeVrio, Myra Cheng
March 2025
-
On the Social and Technical Challenges of Web Search Autosuggestion Moderation
T. J. Hazen, Alexandra Olteanu, Gabriella Kazai, Fernando Diaz, Michael Golebiewski
July 2020
-
-
-
Generation Probabilities Are Not Enough: Uncertainty Highlighting in AI Code Completions
Helena Vasconcelos, Gagan Bansal, Adam Fourney, Q. Vera Liao, Jennifer Wortman Vaughan
ToCHI | April 2025, Vol 32(1)
-
AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap
Q. Vera Liao, Jennifer Wortman Vaughan
Harvard Data Science Review | February 2024
-
Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations
Valerie Chen, Q. Vera Liao, Jennifer Wortman Vaughan, Gagan Bansal
Proc. ACM Hum.-Comput. Interact. | October 2023, Vol 7
Article 370
-
Sensing Wellbeing in the Workplace, Why and For Whom? Envisioning Impacts with Organizational Stakeholders
Anna Kawakami, Shreya Chowdhary, Shamsi T. Iqbal, Q. Liao, Alexandra Olteanu, J. Suh, Koustuv Saha, Alexandra Olteanu, Jina Suh
Proceedings of the ACM on Human-Computer Interaction | March 2023, Vol 7: pp. 1-33
-
Responsible computing during COVID-19 and beyond
Solon Barocas, Asia J. Biega, Margarita Boyarskaya, Kate Crawford, Hal Daumé III, Miro Dudík, Benjamin Fish, Mary L. Gray, Brent Hecht, Alexandra Olteanu, Forough Poursabzi-Sangdeh, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach, Marion Zepf
Communications of the ACM | July 2021, Vol 67(7): pp. 30-32
https://doi.org/10.1145/3466612
-
Measuring algorithmically infused societies
Claudia Wagner, Markus Strohmaier, Alexandra Olteanu, Emre Kiciman, Noshir Contractor, Tina Eliassi-Rad
Nature | June 2021, Vol 595: pp. 197-204
-
Privacy Norms and Preferences for Photos Posted Online
Roberto Hoyle, Luke Stark, Qatrunnada Ismail, David Crandall, Apu Kapadia, Denise Anthony
ACM Transactions on Computer-Human Interaction | March 2020
-
FACTS-IR: Fairness, Accountability, Confidentiality, Transparency, and Safety in Information Retrieval
Alexandra Olteanu, Jean Garcia-Gathright, Maarten de Rijke, Michael D. Ekstrand
ACM SIGIR Forum | December 2019, Vol 53(2): pp. 20
-
Report on the First HIPstIR Workshop on the Future of Information Retrieval
Laura Dietz, Bhaskar Mitra, Jeremy Pickens, Hana Anber, Sandeep Avula, Asia J. Biega, Adrian Boteanu, Shubham Chatterjee, Jeff Dalton, Shiri Dori-Hacohen, John Foley, Henry Feild, Ben Gamari, Rosie Jones, Pallika Kanani, Sumanta Kashyapi, Widad Machmouchi, Matthew Mitsui, Steve Nole, Alexandre Tachard Passos, Jordan Ramsdell, Adam Roegiest, David Smith, Alessandro Sordoni
SIGIR Forum | December 2019, Vol 53(2): pp. 62-75
-
Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries
Alexandra Olteanu, Carlos Castillo, Fernando Diaz, Emre Kiciman
Frontiers in Big Data | July 2019, Vol 2
-