Microsoft Research Lab – New York City

FATE: Fairness, Accountability, Transparency, and Ethics in AI

Papiers de Collaboration

View by:

- Supporting Industry Computing Researchers in Assessing, Articulating, and Addressing the Potential Negative Societal Impact of Their Work
  
  Wesley Hanwen Deng, Solon Barocas, Jennifer Wortman Vaughan
  
  ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2025) | November 2025
  
  Preprint
- Dehumanizing Machines: Mitigating Anthropomorphic Behaviors in Text Generation Systems
  
  Myra Cheng, Su Lin Blodgett, Alicia DeVrio, Lisa Egede, Alexandra Olteanu
  
  2025 Meeting of the Association for Computational Linguistics | July 2025
  
  SAC Highlight
  
  Publication
- Position: To Make Text-to-Image Models that Work for Marginalized Communities, We Need New Measurement Practices for the Long Tail
  
  Nari Johnson, Hamna ., Deepthi Sudharsan, Theo Holroyd, Samantha Dalal, Siobhan Mackenzie Hall, Jennifer Wortman Vaughan, Daniela Massiceti, Cecily Morrison
  
  July 2025
  
  Preprint Project
- Understanding and Meeting Practitioner Needs When Measuring Representational Harms Caused by LLM-Based Systems
  
  Emma Harvey, Emily Sheng, Su Lin Blodgett, Alex Chouldechova, Jean Garcia-Gathright, Alexandra Olteanu, Hanna Wallach
  
  Findings of the Association for Computational Linguistics: ACL 2025 | July 2025
  
  Publication
- Examining the Expanding Role of Synthetic Data Throughout the AI Development Pipeline
  
  Shivani Kapania, Stephanie Ballard, Alex Kessler, Jennifer Wortman Vaughan
  
  8th ACM Conference on Fairness, Accountability, and Transparency (FAccT 2025) | June 2025
  
  Preprint
- Measuring Machine Learning Harms from Stereotypes: Requires Understanding Who is Being Harmed by Which Errors in What Ways
  
  Angelina Wang, Xuechunzi Bai, Solon Barocas, Su Lin Blodgett
  
  2025 ACM Conference on Fairness, Accountability, and Transparency | June 2025
  
  Publication
- Rigor in AI: Doing Rigorous AI Work Requires a Broader, Responsible AI-Informed Conception of Rigor
  
  Alexandra Olteanu, Su Lin Blodgett, Agathe Balayn, Angelina Wang, Fernando Diaz, Flavio du Pin Calmon, Margaret Mitchell, Michael Ekstrand, Reuben Binns, Solon Barocas
  
  June 2025
  
  Publication
- Canvil: Designerly Adaptation for LLM-Powered User Experiences
  
  K. Feng, Q. Vera Liao, Ziang Xiao, Jennifer Wortman Vaughan, Amy Zhang, David W. McDonald
  
  CHI 2025 | April 2025
  
  DOI
- A Taxonomy of Linguistic Expressions That Contribute To Anthropomorphism of Language Technologies
  
  Alicia DeVrio, Myra Cheng, Lisa Egede, Alexandra Olteanu, Su Lin Blodgett
  
  CHI 2025 | April 2025
  
  Publication
- Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistencies
  
  Sunnie S. Y. Kim, Jennifer Wortman Vaughan, Q. Vera Liao, Tania Lombrozo, Olga Russakovsky
  
  CHI 2025 | April 2025
  
  Honorable Mention, CHI 2025
  
  Publication
- “I Am the One and Only, Your Cyber BFF”: Understanding the Impact of GenAI Requires Understanding the Impact of Anthropomorphic AI
  
  Myra Cheng, Alicia DeVrio, Lisa Egede, Su Lin Blodgett, Alexandra Olteanu
  
  April 2025
  
  Publication Publication
- Generation Probabilities Are Not Enough: Uncertainty Highlighting in AI Code Completions
  
  Helena Vasconcelos, Gagan Bansal, Adam Fourney, Q. Vera Liao, Jennifer Wortman Vaughan
  
  ToCHI | April 2025, Vol 32(1)
  
  Publication
- Taxonomizing Representational Harms using Speech Act Theory
  
  Emily Corvi, Hannah Washington, Stefanie Reed, Chad Atalla, Alex Chouldechova, Alex Dow, Jean Garcia-Gathright, Nick Pangakis, Emily Sheng, Dan Vann, Matthew Vogel, Hanna Wallach
  
  March 2025
  
  Publication
- AI Automatons: AI Systems Intended to Imitate Humans
  
  Alexandra Olteanu, Solon Barocas, Su Lin Blodgett, Lisa Egede, Alicia DeVrio, Myra Cheng
  
  March 2025
  
  Preprint
- Position: Evaluating Generative AI Systems is a Social Science Measurement Challenge
  
  Hanna Wallach, Meera Desai, A. Feder Cooper, Angelina Wang, Chad Atalla, Solon Barocas, Su Lin Blodgett, Alex Chouldechova, Emily Corvi, Alex Dow, Jean Garcia-Gathright, Alexandra Olteanu, Nick Pangakis, Stefanie Reed, Emily Sheng, Dan Vann, Jennifer Wortman Vaughan, Matthew Vogel, Hannah Washington, Abigail Z. Jacobs
  
  ICML 2025 | January 2025
  
  Publication
- A Shared Standard for Valid Measurement of Generative AI Systems’ Capabilities, Risks, and Impacts
  
  Alex Chouldechova, Chad Atalla, Solon Barocas, A. Feder Cooper, Emily Corvi, Alex Dow, Jean Garcia-Gathright, Nick Pangakis, Stefanie Reed, Emily Sheng, Dan Vann, Matthew Vogel, Hannah Washington, Hanna Wallach
  
  December 2024
  
  Publication
- Challenges in Human-Agent Communication
  
  Gagan Bansal, Jennifer Wortman Vaughan, Saleema Amershi, Eric Horvitz, Adam Fourney, Hussein Mozannar, Victor Dibia, Daniel S. Weld
  
  MSR-TR-2024-53 | December 2024
  
  Published by Microsoft
  
  Publication Project
- Gaps Between Research and Practice When Measuring Representational Harms Caused by LLM-Based Systems
  
  Emma Harvey, Emily Sheng, Su Lin Blodgett, Alex Chouldechova, Jean Garcia-Gathright, Alexandra Olteanu, Hanna Wallach
  
  November 2024
  
  Publication
- Dimensions of Generative AI Evaluation Design
  
  Alex Dow, Jennifer Wortman Vaughan, Solon Barocas, Chad Atalla, Alex Chouldechova, Hanna Wallach
  
  November 2024
  
  Publication
- Evaluating Generative AI Systems is a Social Science Measurement Challenge
  
  Hanna Wallach, Meera Desai, Nick Pangakis, A. Feder Cooper, Angelina Wang, Solon Barocas, Alex Chouldechova, Chad Atalla, Su Lin Blodgett, Emily Corvi, Alex Dow, Jean Garcia-Gathright, Alexandra Olteanu, Stefanie Reed, Emily Sheng, Dan Vann, Jennifer Wortman Vaughan, Matthew Vogel, Hannah Washington, Abigail Z. Jacobs
  
  November 2024
  
  Publication
- (De)Noise: Moderating the Inconsistency Between Human Decision-Makers
  
  Nina Grgić-Hlača, Junaid Ali, Krishna P. Gummadi, Jennifer Wortman Vaughan
  
  ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2024) | November 2024
  
  Publication
- Metrics for What, Metrics for Whom: Assessing Actionability of Bias Evaluation Metrics in NLP
  
  Pieter Delobelle, Giuseppe Attanasio, Debora Nozza, Su Lin Blodgett, Zeerak Talat
  
  2024 Empirical Methods in Natural Language Processing | November 2024
  
  Publication
- UX Matters: The Critical Role of UX in Responsible AI
  
  Q. Vera Liao, Mihaela Vorvoreanu, Hariharan Subramonyam, Lauren Wilcox
  
  ACM Interactions | July 2024, Vol 31: pp. 22-27
  
  DOI Publication
- “I’m Not Sure, But…”: Examining the Impact of Large Language Models’ Uncertainty Expression on User Reliance and Trust
  
  Sunnie S. Y. Kim, Q. Vera Liao, Mihaela Vorvoreanu, Stephanie Ballard, Jennifer Wortman Vaughan
  
  7th ACM Conference on Fairness, Accountability, and Transparency (FAccT 2024) | June 2024
  
  Publication Project
- The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels
  
  Eve Fleisig, Su Lin Blodgett, Dan Klein, Zeerak Talat
  
  NAACL 2024 | June 2024
  
  Publication
- “One-Size-Fits-All”? Examining Expectations around What Constitute “Fair” or “Good” NLG System Behaviors
  
  Li Lucy, Su Lin Blodgett, Milad Shokouhi, Hanna Wallach, Alexandra Olteanu
  
  NAACL 2024 | June 2024
  
  Publication
- The CLeAR Documentation Framework for AI Transparency: Recommendations for Practitioners & Context for Policymakers
  
  Kasia Chmielinski, Sarah Newman, Chris N. Kranzinger, Michael Hind, Jennifer Wortman Vaughan, Margaret Mitchell, Julia Stoyanovich, Angelina McMillan-Major, Emily McReynolds, Kathleen Esfahany, Mary L. Gray, Audrey Chang, Maui Hudson
  
  April 2024
  
  Publication Project
- Tinker, Tailor, Configure, Customize: The Articulation Work of Customizing AI Fairness Checklists
  
  Michael Madaio, Jingya Chen, Hanna Wallach, Jennifer Wortman Vaughan
  
  Computer-Supported Cooperative Work | April 2024
  
  Publication Project
- Un cadre pour explorer les conséquences de l'accès aux connaissances d'entreprise via l'IA et identifier les risques pour les travailleurs
  
  Anna Gausen, Bhaskar Mitra, Siân Lindley
  
  ACM Conference on Fairness, Accountability, and Transparency | March 2024
  
  Publication Preprint PDF Project Project
- AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap
  
  Q. Vera Liao, Jennifer Wortman Vaughan
  
  Harvard Data Science Review | February 2024
  
  Publication Project
- Through the Looking-Glass: Transparency Implications and Challenges in Enterprise AI Knowledge Systems
  
  Karina Cortiñas-Lorenzo, Siân Lindley, Ida Larsen-Ledet, Bhaskar Mitra
  
  January 2024
  
  Preprint Project Project
- Responsible AI Considerations in Text Summarization Research: A Review of Current Practices
  
  Yu Lu Liu, Meng Cao, Su Lin Blodgett, Jackie Chi Kit Cheung, Alexandra Olteanu, Adam Trischler
  
  Findings of EMNLP 2023 | December 2023
  
  Publication
- A Framework for Automated Measurement of Responsible AI Harms in Generative AI Applications
  
  Ahmed Magooda, Alec Helyar, Kyle Jackson, David Sullivan, Chad Atalla, Emily Sheng, Dan Vann, Richard Edgar, Hamid Palangi, Roman Lutz, Hongliang Kong, Vincent Yun, Eslam Kamal, Federico Zarfati, Hanna Wallach, Sarah Bird, Mei Chen
  
  ArXiv | October 2023, Vol abs/2310.17750
  
  DOI Publication Publication
- Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations
  
  Valerie Chen, Q. Vera Liao, Jennifer Wortman Vaughan, Gagan Bansal
  
  Proc. ACM Hum.-Comput. Interact. | October 2023, Vol 7
  
  Article 370
  
  PDF Project
- Patterns of gender-specializing query reformulation
  
  Amifa Raj, Bhaskar Mitra, Nick Craswell, Michael D. Ekstrand
  
  2023 International ACM SIGIR Conference on Research and Development in Information Retrieval | July 2023
  
  Publication Preprint PDF PDF
- FairPrism: Evaluating Fairness-Related Harms in Text Generation
  
  Eve Fleisig, Aubrie Amstutz, Chad Atalla, Su Lin Blodgett, Hal Daumé III, Alexandra Olteanu, Emily Sheng, Dan Vann, Hanna Wallach
  
  ACL 2023 | July 2023
  
  Publication
- This Prompt is Measuring : Evaluating Bias Evaluation in Language Models
  
  Seraphina Goldfarb-Tarrant, Eddie Ungless, Esma Balkir, Su Lin Blodgett
  
  Findings of the Association for Computational Linguistics: ACL 2023 | July 2023
  
  Publication
- Designerly Understanding: Information Needs for Model Transparency to Support Design Ideation for AI-Powered User Experience
  
  Q. Vera Liao, Hariharan Subramonyam, Jennifer Wang, Jennifer Wortman Vaughan
  
  2023 ACM CHI Conference on Human Factors in Computing Systems (CHI 2023) | April 2023
  
  PDF Project
- fAIlureNotes: Supporting Designers in Understanding the Limits of AI Models for Computer Vision Tasks
  
  Steven Moore, Q. Vera Liao, Hariharan Subramonyam
  
  CHI 2023 | April 2023
  
  Preprint
- GAM Coach: Towards Interactive and User-Centered Algorithmic Recourse
  
  Zijie J. Wang, Jennifer Wortman Vaughan, Rich Caruana, Duen Horng Chau
  
  2023 ACM CHI Conference on Human Factors in Computing Systems (CHI 2023) | February 2023
  
  PDF Project
- Taxonomizing and Measuring Representational Harms: A Look at Image Tagging
  
  Jared Katzman, Angelina Wang, Morgan Scheuerman, Su Lin Blodgett, Kristen Laird, Hanna Wallach, Solon Barocas
  
  AAAI 2022 | February 2023
  
  Publication
- De-Biasing Relevance Judgements for Fair Ranking
  
  Amin Bigdeli, Negar Arabzadeh, Shirin Seyedsalehi, Bhaskar Mitra, Morteza Zihayat, Ebrahim Bagheri
  
  Proceedings of the 45th European Conference on Information Retrieval (ECIR) | January 2023
  
  Publication PDF PDF
- Understanding Machine Learning Practitioners’ Data Documentation Perceptions, Needs, Challenges, and Desiderata
  
  Amy K. Heger, Liz B. Marquis, Mihaela Vorvoreanu, Hanna Wallach, Jennifer Wortman Vaughan
  
  2022 Conference on Computer Supported Cooperative Work | November 2022
  
  Publication Project Project
- Generation Probabilities are Not Enough: Improving Error Highlighting for AI Code Suggestions
  
  Helena Vasconcelos, Gagan Bansal, Adam Fourney, Q. Vera Liao, Jennifer Wortman Vaughan
  
  NeurIPS Workshop on Human-Centered AI | October 2022
  
  Publication
- Connecting Algorithmic Research and Usage Contexts: A Perspective of Contextualized Evaluation for Explainable AI
  
  Q. Vera Liao, Yunfeng Zhang, Ronny Luss, Finale Doshi-Velez, Amit Dhurandhar
  
  AAAI Conference on Human Computation and Crowdsourcing | October 2022
  
  Best Paper Award, Honorable Mention
  
  Publication Project
- Ethical and Social Considerations in Automatic Expert Identification and People Recommendation in Organizational Knowledge Management Systems
  
  Ida Larsen-Ledet, Bhaskar Mitra, Siân Lindley
  
  Proceedings of the 5th FAccTRec Workshop on Responsible Recommendation at RecSys 2022 | September 2022
  
  Preprint PDF Project Project
- Examining Responsibility and Deliberation in AI Impact Statements and Ethics Reviews
  
  David Liu, Priyanka Nanayakkara, Sarah Ariyan Sakha, Grace Abuhamad, Su Lin Blodgett, Nicholas Diakopoulos, Jessica R. Hullman, Tina Eliassi-Rad
  
  AIES 2022 | July 2022
  
  Publication
- Interpretability, Then What? Editing Machine Learning Models to Reflect Human Knowledge and Values
  
  Zijie J. Wang, Alex Kale, Harsha Nori, Peter Stella, Mark E. Nunnally, Duen Horng Chau, Mihaela Vorvoreanu, Jennifer Wortman Vaughan, Rich Caruana
  
  KDD 2022 | July 2022
  
  Publication Project
- How Different Groups Prioritize Ethical Values for Responsible AI
  
  Maurice Jakesch, Zana Buçinca, Saleema Amershi, Alexandra Olteanu
  
  FAccT 2022 | May 2022
  
  Publication DOI
- Deconstructing NLG Evaluation: Evaluation Practices, Assumptions, and Their Implications
  
  Kaitlyn Zhou, Su Lin Blodgett, Adam Trischler, Hal Daumé III, Kaheer Suleman, Alexandra Olteanu
  
  NAACL 2022 | May 2022
  
  Publication
- REAL ML: Recognizing, Exploring, and Articulating Limitations of Machine Learning Research
  
  Jessie J. Smith, Saleema Amershi, Solon Barocas, Hanna Wallach, Jennifer Wortman Vaughan
  
  FAccT 2022 | May 2022
  
  DOI Project
- Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation
  
  Vivian Lai, Samuel Carton, Rajat Bhatnagar, Q. Vera Liao, Yunfeng Zhang, Chenhao Tan
  
  CHI 2022 | May 2022
  
  Publication
- Designing for Responsible Trust in AI Systems: A Communication Perspective
  
  Q. Vera Liao, S. Shyam Sundar
  
  FAccT 2022 | April 2022
  
  Preprint
  
  Publication
- Joint Multisided Exposure Fairness for Recommendation
  
  Bhaskar Mitra, Haolun Wu, Chen Ma, Fernando Diaz, Xue Liu
  
  2022 International ACM SIGIR Conference on Research and Development in Information Retrieval | April 2022
  
  Publication PDF PDF
- Responsible Language Technologies: Foreseeing and Mitigating Harms
  
  Su Lin Blodgett, Q. Vera Liao, Alexandra Olteanu, Rada Mihalcea, Michael Muller, Morgan Klaus Scheuerman, Chenhao Tan, Qian Yang
  
  CHI 2022 | April 2022
  
  Publication
- Assessing the Fairness of AI Systems: AI Practitioners’ Processes, Challenges, and Needs for Support
  
  Michael Madaio, Lisa Egede, Hariharan Subramonyam, Jennifer Wortman Vaughan, Hanna Wallach
  
  25th ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2022) | February 2022
  
  Publication
- Bias-aware Fair Neural Ranking for Addressing Stereotypical Gender Biases
  
  Shirin SeyedSalehi, Amin Bigdeli, Negar Arabzadeh, Bhaskar Mitra, Morteza Zihayat, Ebrahim Bagheri
  
  Proceedings of the 25th International Conference on Extending Database Technology (EDBT) | February 2022
  
  Publication PDF PDF
- Exposing Query Identification for Search Transparency
  
  Ruohan Li, Jianxiang Li, Bhaskar Mitra, Fernando Diaz, Asia Biega
  
  2022 The Web Conference | January 2022
  
  Publication Preprint PDF PDF
- A Human-Centered Agenda for Intelligible Machine Learning
  
  Jennifer Wortman Vaughan, Hanna Wallach
  
  In Machines We Trust: Perspectives on Dependable AI | Published by MIT Press | 2021
  
  Author's Version Project
- Datasheets for Datasets
  
  Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, Kate Crawford
  
  Communications of the ACM | December 2021, Vol 64(12): pp. 86-92
  
  Publication Project Project
- Summarize with Caution: Comparing Global Feature Attributions
  
  Alex Okeson, Rich Caruana, Nick Craswell, Kori Inkpen, Scott M. Lundberg, Harsha Nori, Hanna Wallach, Jennifer Wortman Vaughan
  
  Bulletin of the IEEE Computer Society Technical Committee on Data Engineering | December 2021
  
  Publication Project
- Revisiting Popularity and Demographic Biases in Recommender Evaluation and Effectiveness
  
  Nicola Neophytou, Bhaskar Mitra, Catherine Stinson
  
  European Conference on Information Retrieval | November 2021
  
  Publication Preprint PDF PDF
- Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets
  
  Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, Hanna Wallach
  
  ACL-IJCNLP 2021 | August 2021
  
  Publication
- Responsible computing during COVID-19 and beyond
  
  Solon Barocas, Asia J. Biega, Margarita Boyarskaya, Kate Crawford, Hal Daumé III, Miro Dudík, Benjamin Fish, Mary L. Gray, Brent Hecht, Alexandra Olteanu, Forough Poursabzi-Sangdeh, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach, Marion Zepf
  
  Communications of the ACM | July 2021, Vol 67(7): pp. 30-32
  
  https://doi.org/10.1145/3466612
  
  Publication
- Measuring algorithmically infused societies
  
  Claudia Wagner, Markus Strohmaier, Alexandra Olteanu, Emre Kiciman, Noshir Contractor, Tina Eliassi-Rad
  
  Nature | June 2021, Vol 595: pp. 197-204
  
  Publication Publication
- Manipulating and Measuring Model Interpretability
  
  Forough Poursabzi-Sangdeh, Daniel G. Goldstein, Jake Hofman, Jennifer Wortman Vaughan, Hanna Wallach
  
  CHI 2021 | May 2021
  
  Publication Project
- “I Can’t Reply with That”: Characterizing Problematic Email Reply Suggestions
  
  Ronald Robertson, Alexandra Olteanu, Fernando Diaz, Milad Shokouhi, Peter Bailey
  
  CHI Conference on Human Factors in Computing Systems (CHI ’21) | May 2021
  
  Publication Video
- Designing Disaggregated Evaluations of AI Systems: Choices, Considerations, and Tradeoffs
  
  Solon Barocas, Anhong Guo, Ece Kamar, Jacquelyn Krones, Meredith Ringel Morris, Jennifer Wortman Vaughan, Duncan Wadsworth, Hanna Wallach
  
  AIES 2021 | May 2021
  
  Publication
- Measurement and Fairness
  
  Abigail Z. Jacobs, Hanna Wallach
  
  Fairness, Accountability, and Transparency (FAccT ’21) | March 2021
  
  Publication DOI
- Overcoming Failures of Imagination in AI Infused System Development and Deployment
  
  Margarita Boyarskaya, Alexandra Olteanu, Kate Crawford
  
  In the Navigating the Broader Impacts of AI Research Workshop at NeurIPS 2020 | December 2020
  
  Publication Video
- Assessing and mitigating unfairness in credit models with the Fairlearn toolkit
  
  Miro Dudík, William Chen, Solon Barocas, Mario Inchiosa, Nick Lewins, Miruna Oprescu, Joy Qiao, Mehrnoosh Sameki, Mario Schlener, Jason Tuo, Hanna Wallach
  
  MSR-TR-2020-34 | September 2020
  
  Published by Microsoft
  
  Publication
- Mitigating bias in algorithmic hiring: evaluating claims and practices
  
  Manish Raghavan, Solon Barocas, Jon Kleinberg, Karen Levy
  
  FAT* 2020 | August 2020
  
  Publication
- The Hidden Assumptions Behind Counterfactual Explanations and Principal Reasons
  
  Solon Barocas, Andrew Selbst, Manish Raghavan
  
  FAT* 2020 | August 2020
  
  Publication Project
- Quand les suggestions de fin de recherche sont-elles problématiques ?
  
  Alexandra Olteanu, Fernando Diaz, Gabriella Kazai
  
  Computer Supported Collaborative Work and Social Computing (CSCW) | August 2020
  
  CSCW 2020 Honorable Mention
  
  CSCW 2020 Honorable Mention
  
  Publication Video
- Operationalizing the Legal Principle of Data Minimization for Personalization
  
  Asia J. Biega, Peter Potash, Hal Daumé III, Fernando Diaz, Michèle Finck
  
  SIGIR 2020 | July 2020
  
  PDF
- Evaluating Stochastic Rankings with Expected Exposure
  
  Fernando Diaz, Bhaskar Mitra, Michael D. Ekstrand, Asia J. Biega, Ben Carterette
  
  Proceedings of the 29th ACM International Conference on Information and Knowledge Management (CIKM) | July 2020
  
  Best Long Research Paper Nominee
  
  Publication Preprint PDF
- On the Social and Technical Challenges of Web Search Autosuggestion Moderation
  
  T. J. Hazen, Alexandra Olteanu, Gabriella Kazai, Fernando Diaz, Michael Golebiewski
  
  July 2020
  
  Publication
- Language (Technology) is Power: A Critical Survey of “Bias” in NLP
  
  Su Lin Blodgett, Solon Barocas, Hal Daumé III, Hanna Wallach
  
  ACL | June 2020
  
  Publication Video
- Privacy Dependencies
  
  Solon Barocas, Karen Levy
  
  Washington Law Review | June 2020, Vol 95(2): pp. 555-615
  
  Publication
- Fairlearn: A toolkit for assessing and improving fairness in AI
  
  Sarah Bird, Miro Dudík, Richard Edgar, Brandon Horn, Roman Lutz, Vanessa Milan, Mehrnoosh Sameki, Hanna Wallach, Kathleen Walker
  
  MSR-TR-2020-32 | May 2020
  
  Published by Microsoft
  
  Publication
- Towards Query Logs For Privacy Studies: On Deriving Search Queries From Questions
  
  Asia J. Biega, Jana Schmidt, Rishiraj Saha Roy
  
  ECIR 2020 | April 2020
  
  Publication
- A Human in the Loop is Not Enough: The Need for Human-Subject Experiments in Facial Recognition
  
  Forough Poursabzi-Sangdeh, Samira Samadi, Jennifer Wortman Vaughan, Hanna Wallach
  
  April 2020
  
  PDF
- Interpreting Interpretability: Understanding Data Scientists’ Use of Interpretability Tools for Machine Learning
  
  Harmanpreet Kaur, Harsha Nori, Samuel Jenkins, Rich Caruana, Hanna Wallach, Jennifer Wortman Vaughan
  
  CHI 2020 | April 2020
  
  CHI 2020 Honorable Mention Award
  
  Publication Github Project
- “I Don’t Want Someone to Watch Me While I’m Working”: Gendered Views of Facial Recognition Technology in Workplace Surveillance
  
  Luke Stark, Amanda Stanhaus, Denise L. Anthony
  
  Journal of the Association for Information Science and Technology | March 2020
  
  Publication
- Co-Designing Checklists to Understand Organizational Challenges and Opportunities around Fairness in AI
  
  Michael Madaio, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach
  
  CHI Conference on Human Factors in Computing Systems | March 2020
  
  CHI 2020 Best Paper Award
  
  PDF Project
- Roles for computing in social change
  
  Rediet Abebe, Solon Barocas, Jon Kleinberg, Karen Levy, Manish Raghavan, David Robinson
  
  FAT* | January 2020
  
  Publication
- When not to design, build, or deploy
  
  Solon Barocas, Asia J. Biega, Benjamin Fish, Jędrzej Niklas, Luke Stark
  
  2020 Conference on Fairness, Accountability, and Transparency (FAT* ’20) | January 2020
  
  Publication Author's Version
- FACTS-IR: Fairness, Accountability, Confidentiality, Transparency, and Safety in Information Retrieval
  
  Alexandra Olteanu, Jean Garcia-Gathright, Maarten de Rijke, Michael D. Ekstrand
  
  ACM SIGIR Forum | December 2019, Vol 53(2): pp. 20
  
  Publication
- Stretching Human Laws to Apply to Machines: The Dangers of a ‘Colorblind’ Computer
  
  Zach Harned, Hanna Wallach
  
  Florida State University Law Review, Forthcoming | December 2019
  
  Publication
- Overview of the TREC 2019 Fair Ranking Track
  
  Asia J. Biega, Fernando Diaz, Michael D. Ekstrand, Sebastian Kohlmeier
  
  TREC 2019 | November 2019
  
  Publication Publication
- Toward Fairness in AI for People with Disabilities: A Research Roadmap
  
  Anhong Guo, Ece Kamar, Jennifer Wortman Vaughan, Hanna Wallach, Meredith Ringel Morris
  
  ASSETS 2019 Workshop on AI Fairness for People with Disabilities | October 2019
  
  Publication Video Project
- Fair Regression: Quantitative Definitions and Reduction-Based Algorithms
  
  Alekh Agarwal, Miro Dudík, Zhiwei Steven Wu
  
  36th International Conference on Machine Learning | September 2019
  
  PDF Github
- Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries
  
  Alexandra Olteanu, Carlos Castillo, Fernando Diaz, Emre Kiciman
  
  Frontiers in Big Data | July 2019, Vol 2
  
  DOI Publication
- Improving fairness in machine learning systems: What do industry practitioners need?
  
  Ken Holstein, Jennifer Wortman Vaughan, Hal Daumé III, Miro Dudík, Hanna Wallach
  
  2019 ACM CHI Conference on Human Factors in Computing Systems | June 2019
  
  Publication
- Understanding the Effect of Accuracy on Trust in Machine Learning Models
  
  Ming Ying, Jennifer Wortman Vaughan, Hanna Wallach
  
  2019 ACM CHI Conference on Human Factors in Computing Systems | May 2019
  
  Honorable Mention Award at CHI 2019
  
  Publication Project
- What’s in a Name? Reducing Bias in Bios without Access to Protected Attributes
  
  Alexey Romanov, Maria De-Arteaga, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alex Chouldechova, Sahin Geyik, krishnaram Kenthapadi, Anna Rumshisky, Adam Tauman Kalai
  
  NAACL 2019 | April 2019
  
  Best Paper Award, NAACL 2019
  
  Publication
- The Disparate Effects of Strategic Manipulation
  
  Lily Hu, Nicole Immorlica, Jennifer Wortman Vaughan
  
  2nd ACM Conference on Fairness, Accountability, and Transparency | January 2019
  
  Publication
- The Externalities of Exploration and How Data Diversity Helps Exploitation
  
  Manish Raghavan, Aleksandrs Slivkins, Jennifer Wortman Vaughan, Zhiwei Steven Wu
  
  31st Annual Conference on Learning Theory | February 2018
  
  Publication

- Supporting Industry Computing Researchers in Assessing, Articulating, and Addressing the Potential Negative Societal Impact of Their Work
  
  Wesley Hanwen Deng, Solon Barocas, Jennifer Wortman Vaughan
  
  ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2025) | November 2025
  
  Preprint
- Position: To Make Text-to-Image Models that Work for Marginalized Communities, We Need New Measurement Practices for the Long Tail
  
  Nari Johnson, Hamna ., Deepthi Sudharsan, Theo Holroyd, Samantha Dalal, Siobhan Mackenzie Hall, Jennifer Wortman Vaughan, Daniela Massiceti, Cecily Morrison
  
  July 2025
  
  Preprint Project
- Rigor in AI: Doing Rigorous AI Work Requires a Broader, Responsible AI-Informed Conception of Rigor
  
  Alexandra Olteanu, Su Lin Blodgett, Agathe Balayn, Angelina Wang, Fernando Diaz, Flavio du Pin Calmon, Margaret Mitchell, Michael Ekstrand, Reuben Binns, Solon Barocas
  
  June 2025
  
  Publication
- Examining the Expanding Role of Synthetic Data Throughout the AI Development Pipeline
  
  Shivani Kapania, Stephanie Ballard, Alex Kessler, Jennifer Wortman Vaughan
  
  8th ACM Conference on Fairness, Accountability, and Transparency (FAccT 2025) | June 2025
  
  Preprint
- Canvil: Designerly Adaptation for LLM-Powered User Experiences
  
  K. Feng, Q. Vera Liao, Ziang Xiao, Jennifer Wortman Vaughan, Amy Zhang, David W. McDonald
  
  CHI 2025 | April 2025
  
  DOI
- Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistencies
  
  Sunnie S. Y. Kim, Jennifer Wortman Vaughan, Q. Vera Liao, Tania Lombrozo, Olga Russakovsky
  
  CHI 2025 | April 2025
  
  Honorable Mention, CHI 2025
  
  Publication
- A Taxonomy of Linguistic Expressions That Contribute To Anthropomorphism of Language Technologies
  
  Alicia DeVrio, Myra Cheng, Lisa Egede, Alexandra Olteanu, Su Lin Blodgett
  
  CHI 2025 | April 2025
  
  Publication
- “I Am the One and Only, Your Cyber BFF”: Understanding the Impact of GenAI Requires Understanding the Impact of Anthropomorphic AI
  
  Myra Cheng, Alicia DeVrio, Lisa Egede, Su Lin Blodgett, Alexandra Olteanu
  
  April 2025
  
  Publication Publication
- Generation Probabilities Are Not Enough: Uncertainty Highlighting in AI Code Completions
  
  Helena Vasconcelos, Gagan Bansal, Adam Fourney, Q. Vera Liao, Jennifer Wortman Vaughan
  
  ToCHI | April 2025, Vol 32(1)
  
  Publication
- Taxonomizing Representational Harms using Speech Act Theory
  
  Emily Corvi, Hannah Washington, Stefanie Reed, Chad Atalla, Alex Chouldechova, Alex Dow, Jean Garcia-Gathright, Nick Pangakis, Emily Sheng, Dan Vann, Matthew Vogel, Hanna Wallach
  
  March 2025
  
  Publication
- Position: Evaluating Generative AI Systems is a Social Science Measurement Challenge
  
  Hanna Wallach, Meera Desai, A. Feder Cooper, Angelina Wang, Chad Atalla, Solon Barocas, Su Lin Blodgett, Alex Chouldechova, Emily Corvi, Alex Dow, Jean Garcia-Gathright, Alexandra Olteanu, Nick Pangakis, Stefanie Reed, Emily Sheng, Dan Vann, Jennifer Wortman Vaughan, Matthew Vogel, Hannah Washington, Abigail Z. Jacobs
  
  ICML 2025 | January 2025
  
  Publication
- A Shared Standard for Valid Measurement of Generative AI Systems’ Capabilities, Risks, and Impacts
  
  Alex Chouldechova, Chad Atalla, Solon Barocas, A. Feder Cooper, Emily Corvi, Alex Dow, Jean Garcia-Gathright, Nick Pangakis, Stefanie Reed, Emily Sheng, Dan Vann, Matthew Vogel, Hannah Washington, Hanna Wallach
  
  December 2024
  
  Publication
- Challenges in Human-Agent Communication
  
  Gagan Bansal, Jennifer Wortman Vaughan, Saleema Amershi, Eric Horvitz, Adam Fourney, Hussein Mozannar, Victor Dibia, Daniel S. Weld
  
  MSR-TR-2024-53 | December 2024
  
  Published by Microsoft
  
  Publication Project
- Gaps Between Research and Practice When Measuring Representational Harms Caused by LLM-Based Systems
  
  Emma Harvey, Emily Sheng, Su Lin Blodgett, Alex Chouldechova, Jean Garcia-Gathright, Alexandra Olteanu, Hanna Wallach
  
  November 2024
  
  Publication
- Dimensions of Generative AI Evaluation Design
  
  Alex Dow, Jennifer Wortman Vaughan, Solon Barocas, Chad Atalla, Alex Chouldechova, Hanna Wallach
  
  November 2024
  
  Publication
- Evaluating Generative AI Systems is a Social Science Measurement Challenge
  
  Hanna Wallach, Meera Desai, Nick Pangakis, A. Feder Cooper, Angelina Wang, Solon Barocas, Alex Chouldechova, Chad Atalla, Su Lin Blodgett, Emily Corvi, Alex Dow, Jean Garcia-Gathright, Alexandra Olteanu, Stefanie Reed, Emily Sheng, Dan Vann, Jennifer Wortman Vaughan, Matthew Vogel, Hannah Washington, Abigail Z. Jacobs
  
  November 2024
  
  Publication
- (De)Noise: Moderating the Inconsistency Between Human Decision-Makers
  
  Nina Grgić-Hlača, Junaid Ali, Krishna P. Gummadi, Jennifer Wortman Vaughan
  
  ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2024) | November 2024
  
  Publication
- UX Matters: The Critical Role of UX in Responsible AI
  
  Q. Vera Liao, Mihaela Vorvoreanu, Hariharan Subramonyam, Lauren Wilcox
  
  ACM Interactions | July 2024, Vol 31: pp. 22-27
  
  DOI Publication
- “I’m Not Sure, But…”: Examining the Impact of Large Language Models’ Uncertainty Expression on User Reliance and Trust
  
  Sunnie S. Y. Kim, Q. Vera Liao, Mihaela Vorvoreanu, Stephanie Ballard, Jennifer Wortman Vaughan
  
  7th ACM Conference on Fairness, Accountability, and Transparency (FAccT 2024) | June 2024
  
  Publication Project
- The CLeAR Documentation Framework for AI Transparency: Recommendations for Practitioners & Context for Policymakers
  
  Kasia Chmielinski, Sarah Newman, Chris N. Kranzinger, Michael Hind, Jennifer Wortman Vaughan, Margaret Mitchell, Julia Stoyanovich, Angelina McMillan-Major, Emily McReynolds, Kathleen Esfahany, Mary L. Gray, Audrey Chang, Maui Hudson
  
  April 2024
  
  Publication Project
- Tinker, Tailor, Configure, Customize: The Articulation Work of Customizing AI Fairness Checklists
  
  Michael Madaio, Jingya Chen, Hanna Wallach, Jennifer Wortman Vaughan
  
  Computer-Supported Cooperative Work | April 2024
  
  Publication Project
- Un cadre pour explorer les conséquences de l'accès aux connaissances d'entreprise via l'IA et identifier les risques pour les travailleurs
  
  Anna Gausen, Bhaskar Mitra, Siân Lindley
  
  ACM Conference on Fairness, Accountability, and Transparency | March 2024
  
  Publication Preprint PDF Project Project
- AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap
  
  Q. Vera Liao, Jennifer Wortman Vaughan
  
  Harvard Data Science Review | February 2024
  
  Publication Project
- Through the Looking-Glass: Transparency Implications and Challenges in Enterprise AI Knowledge Systems
  
  Karina Cortiñas-Lorenzo, Siân Lindley, Ida Larsen-Ledet, Bhaskar Mitra
  
  January 2024
  
  Preprint Project Project
- A Framework for Automated Measurement of Responsible AI Harms in Generative AI Applications
  
  Ahmed Magooda, Alec Helyar, Kyle Jackson, David Sullivan, Chad Atalla, Emily Sheng, Dan Vann, Richard Edgar, Hamid Palangi, Roman Lutz, Hongliang Kong, Vincent Yun, Eslam Kamal, Federico Zarfati, Hanna Wallach, Sarah Bird, Mei Chen
  
  ArXiv | October 2023, Vol abs/2310.17750
  
  DOI Publication Publication
- Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations
  
  Valerie Chen, Q. Vera Liao, Jennifer Wortman Vaughan, Gagan Bansal
  
  Proc. ACM Hum.-Comput. Interact. | October 2023, Vol 7
  
  Article 370
  
  PDF Project
- FairPrism: Evaluating Fairness-Related Harms in Text Generation
  
  Eve Fleisig, Aubrie Amstutz, Chad Atalla, Su Lin Blodgett, Hal Daumé III, Alexandra Olteanu, Emily Sheng, Dan Vann, Hanna Wallach
  
  ACL 2023 | July 2023
  
  Publication
- Designerly Understanding: Information Needs for Model Transparency to Support Design Ideation for AI-Powered User Experience
  
  Q. Vera Liao, Hariharan Subramonyam, Jennifer Wang, Jennifer Wortman Vaughan
  
  2023 ACM CHI Conference on Human Factors in Computing Systems (CHI 2023) | April 2023
  
  PDF Project
- Taxonomizing and Measuring Representational Harms: A Look at Image Tagging
  
  Jared Katzman, Angelina Wang, Morgan Scheuerman, Su Lin Blodgett, Kristen Laird, Hanna Wallach, Solon Barocas
  
  AAAI 2022 | February 2023
  
  Publication
- GAM Coach: Towards Interactive and User-Centered Algorithmic Recourse
  
  Zijie J. Wang, Jennifer Wortman Vaughan, Rich Caruana, Duen Horng Chau
  
  2023 ACM CHI Conference on Human Factors in Computing Systems (CHI 2023) | February 2023
  
  PDF Project
- De-Biasing Relevance Judgements for Fair Ranking
  
  Amin Bigdeli, Negar Arabzadeh, Shirin Seyedsalehi, Bhaskar Mitra, Morteza Zihayat, Ebrahim Bagheri
  
  Proceedings of the 45th European Conference on Information Retrieval (ECIR) | January 2023
  
  Publication PDF PDF
- Understanding Machine Learning Practitioners’ Data Documentation Perceptions, Needs, Challenges, and Desiderata
  
  Amy K. Heger, Liz B. Marquis, Mihaela Vorvoreanu, Hanna Wallach, Jennifer Wortman Vaughan
  
  2022 Conference on Computer Supported Cooperative Work | November 2022
  
  Publication Project Project
- Generation Probabilities are Not Enough: Improving Error Highlighting for AI Code Suggestions
  
  Helena Vasconcelos, Gagan Bansal, Adam Fourney, Q. Vera Liao, Jennifer Wortman Vaughan
  
  NeurIPS Workshop on Human-Centered AI | October 2022
  
  Publication
- Connecting Algorithmic Research and Usage Contexts: A Perspective of Contextualized Evaluation for Explainable AI
  
  Q. Vera Liao, Yunfeng Zhang, Ronny Luss, Finale Doshi-Velez, Amit Dhurandhar
  
  AAAI Conference on Human Computation and Crowdsourcing | October 2022
  
  Best Paper Award, Honorable Mention
  
  Publication Project
- Ethical and Social Considerations in Automatic Expert Identification and People Recommendation in Organizational Knowledge Management Systems
  
  Ida Larsen-Ledet, Bhaskar Mitra, Siân Lindley
  
  Proceedings of the 5th FAccTRec Workshop on Responsible Recommendation at RecSys 2022 | September 2022
  
  Preprint PDF Project Project
- Examining Responsibility and Deliberation in AI Impact Statements and Ethics Reviews
  
  David Liu, Priyanka Nanayakkara, Sarah Ariyan Sakha, Grace Abuhamad, Su Lin Blodgett, Nicholas Diakopoulos, Jessica R. Hullman, Tina Eliassi-Rad
  
  AIES 2022 | July 2022
  
  Publication
- Interpretability, Then What? Editing Machine Learning Models to Reflect Human Knowledge and Values
  
  Zijie J. Wang, Alex Kale, Harsha Nori, Peter Stella, Mark E. Nunnally, Duen Horng Chau, Mihaela Vorvoreanu, Jennifer Wortman Vaughan, Rich Caruana
  
  KDD 2022 | July 2022
  
  Publication Project
- How Different Groups Prioritize Ethical Values for Responsible AI
  
  Maurice Jakesch, Zana Buçinca, Saleema Amershi, Alexandra Olteanu
  
  FAccT 2022 | May 2022
  
  Publication DOI
- REAL ML: Recognizing, Exploring, and Articulating Limitations of Machine Learning Research
  
  Jessie J. Smith, Saleema Amershi, Solon Barocas, Hanna Wallach, Jennifer Wortman Vaughan
  
  FAccT 2022 | May 2022
  
  DOI Project
- Designing for Responsible Trust in AI Systems: A Communication Perspective
  
  Q. Vera Liao, S. Shyam Sundar
  
  FAccT 2022 | April 2022
  
  Preprint
  
  Publication
- Joint Multisided Exposure Fairness for Recommendation
  
  Bhaskar Mitra, Haolun Wu, Chen Ma, Fernando Diaz, Xue Liu
  
  2022 International ACM SIGIR Conference on Research and Development in Information Retrieval | April 2022
  
  Publication PDF PDF
- Assessing the Fairness of AI Systems: AI Practitioners’ Processes, Challenges, and Needs for Support
  
  Michael Madaio, Lisa Egede, Hariharan Subramonyam, Jennifer Wortman Vaughan, Hanna Wallach
  
  25th ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2022) | February 2022
  
  Publication
- Bias-aware Fair Neural Ranking for Addressing Stereotypical Gender Biases
  
  Shirin SeyedSalehi, Amin Bigdeli, Negar Arabzadeh, Bhaskar Mitra, Morteza Zihayat, Ebrahim Bagheri
  
  Proceedings of the 25th International Conference on Extending Database Technology (EDBT) | February 2022
  
  Publication PDF PDF
- Exposing Query Identification for Search Transparency
  
  Ruohan Li, Jianxiang Li, Bhaskar Mitra, Fernando Diaz, Asia Biega
  
  2022 The Web Conference | January 2022
  
  Publication Preprint PDF PDF
- A Human-Centered Agenda for Intelligible Machine Learning
  
  Jennifer Wortman Vaughan, Hanna Wallach
  
  In Machines We Trust: Perspectives on Dependable AI | Published by MIT Press | 2021
  
  Author's Version Project
- Summarize with Caution: Comparing Global Feature Attributions
  
  Alex Okeson, Rich Caruana, Nick Craswell, Kori Inkpen, Scott M. Lundberg, Harsha Nori, Hanna Wallach, Jennifer Wortman Vaughan
  
  Bulletin of the IEEE Computer Society Technical Committee on Data Engineering | December 2021
  
  Publication Project
- Datasheets for Datasets
  
  Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, Kate Crawford
  
  Communications of the ACM | December 2021, Vol 64(12): pp. 86-92
  
  Publication Project Project
- Revisiting Popularity and Demographic Biases in Recommender Evaluation and Effectiveness
  
  Nicola Neophytou, Bhaskar Mitra, Catherine Stinson
  
  European Conference on Information Retrieval | November 2021
  
  Publication Preprint PDF PDF
- Responsible computing during COVID-19 and beyond
  
  Solon Barocas, Asia J. Biega, Margarita Boyarskaya, Kate Crawford, Hal Daumé III, Miro Dudík, Benjamin Fish, Mary L. Gray, Brent Hecht, Alexandra Olteanu, Forough Poursabzi-Sangdeh, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach, Marion Zepf
  
  Communications of the ACM | July 2021, Vol 67(7): pp. 30-32
  
  https://doi.org/10.1145/3466612
  
  Publication
- Manipulating and Measuring Model Interpretability
  
  Forough Poursabzi-Sangdeh, Daniel G. Goldstein, Jake Hofman, Jennifer Wortman Vaughan, Hanna Wallach
  
  CHI 2021 | May 2021
  
  Publication Project
- Designing Disaggregated Evaluations of AI Systems: Choices, Considerations, and Tradeoffs
  
  Solon Barocas, Anhong Guo, Ece Kamar, Jacquelyn Krones, Meredith Ringel Morris, Jennifer Wortman Vaughan, Duncan Wadsworth, Hanna Wallach
  
  AIES 2021 | May 2021
  
  Publication
- Measurement and Fairness
  
  Abigail Z. Jacobs, Hanna Wallach
  
  Fairness, Accountability, and Transparency (FAccT ’21) | March 2021
  
  Publication DOI
- Overcoming Failures of Imagination in AI Infused System Development and Deployment
  
  Margarita Boyarskaya, Alexandra Olteanu, Kate Crawford
  
  In the Navigating the Broader Impacts of AI Research Workshop at NeurIPS 2020 | December 2020
  
  Publication Video
- Assessing and mitigating unfairness in credit models with the Fairlearn toolkit
  
  Miro Dudík, William Chen, Solon Barocas, Mario Inchiosa, Nick Lewins, Miruna Oprescu, Joy Qiao, Mehrnoosh Sameki, Mario Schlener, Jason Tuo, Hanna Wallach
  
  MSR-TR-2020-34 | September 2020
  
  Published by Microsoft
  
  Publication
- Mitigating bias in algorithmic hiring: evaluating claims and practices
  
  Manish Raghavan, Solon Barocas, Jon Kleinberg, Karen Levy
  
  FAT* 2020 | August 2020
  
  Publication
- The Hidden Assumptions Behind Counterfactual Explanations and Principal Reasons
  
  Solon Barocas, Andrew Selbst, Manish Raghavan
  
  FAT* 2020 | August 2020
  
  Publication Project
- Operationalizing the Legal Principle of Data Minimization for Personalization
  
  Asia J. Biega, Peter Potash, Hal Daumé III, Fernando Diaz, Michèle Finck
  
  SIGIR 2020 | July 2020
  
  PDF
- Evaluating Stochastic Rankings with Expected Exposure
  
  Fernando Diaz, Bhaskar Mitra, Michael D. Ekstrand, Asia J. Biega, Ben Carterette
  
  Proceedings of the 29th ACM International Conference on Information and Knowledge Management (CIKM) | July 2020
  
  Best Long Research Paper Nominee
  
  Publication Preprint PDF
- Privacy Dependencies
  
  Solon Barocas, Karen Levy
  
  Washington Law Review | June 2020, Vol 95(2): pp. 555-615
  
  Publication
- Language (Technology) is Power: A Critical Survey of “Bias” in NLP
  
  Su Lin Blodgett, Solon Barocas, Hal Daumé III, Hanna Wallach
  
  ACL | June 2020
  
  Publication Video
- Fairlearn: A toolkit for assessing and improving fairness in AI
  
  Sarah Bird, Miro Dudík, Richard Edgar, Brandon Horn, Roman Lutz, Vanessa Milan, Mehrnoosh Sameki, Hanna Wallach, Kathleen Walker
  
  MSR-TR-2020-32 | May 2020
  
  Published by Microsoft
  
  Publication
- Towards Query Logs For Privacy Studies: On Deriving Search Queries From Questions
  
  Asia J. Biega, Jana Schmidt, Rishiraj Saha Roy
  
  ECIR 2020 | April 2020
  
  Publication
- A Human in the Loop is Not Enough: The Need for Human-Subject Experiments in Facial Recognition
  
  Forough Poursabzi-Sangdeh, Samira Samadi, Jennifer Wortman Vaughan, Hanna Wallach
  
  April 2020
  
  PDF
- Interpreting Interpretability: Understanding Data Scientists’ Use of Interpretability Tools for Machine Learning
  
  Harmanpreet Kaur, Harsha Nori, Samuel Jenkins, Rich Caruana, Hanna Wallach, Jennifer Wortman Vaughan
  
  CHI 2020 | April 2020
  
  CHI 2020 Honorable Mention Award
  
  Publication Github Project
- “I Don’t Want Someone to Watch Me While I’m Working”: Gendered Views of Facial Recognition Technology in Workplace Surveillance
  
  Luke Stark, Amanda Stanhaus, Denise L. Anthony
  
  Journal of the Association for Information Science and Technology | March 2020
  
  Publication
- Co-Designing Checklists to Understand Organizational Challenges and Opportunities around Fairness in AI
  
  Michael Madaio, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach
  
  CHI Conference on Human Factors in Computing Systems | March 2020
  
  CHI 2020 Best Paper Award
  
  PDF Project
- When not to design, build, or deploy
  
  Solon Barocas, Asia J. Biega, Benjamin Fish, Jędrzej Niklas, Luke Stark
  
  2020 Conference on Fairness, Accountability, and Transparency (FAT* ’20) | January 2020
  
  Publication Author's Version
- FACTS-IR: Fairness, Accountability, Confidentiality, Transparency, and Safety in Information Retrieval
  
  Alexandra Olteanu, Jean Garcia-Gathright, Maarten de Rijke, Michael D. Ekstrand
  
  ACM SIGIR Forum | December 2019, Vol 53(2): pp. 20
  
  Publication
- Stretching Human Laws to Apply to Machines: The Dangers of a ‘Colorblind’ Computer
  
  Zach Harned, Hanna Wallach
  
  Florida State University Law Review, Forthcoming | December 2019
  
  Publication
- Overview of the TREC 2019 Fair Ranking Track
  
  Asia J. Biega, Fernando Diaz, Michael D. Ekstrand, Sebastian Kohlmeier
  
  TREC 2019 | November 2019
  
  Publication Publication
- Toward Fairness in AI for People with Disabilities: A Research Roadmap
  
  Anhong Guo, Ece Kamar, Jennifer Wortman Vaughan, Hanna Wallach, Meredith Ringel Morris
  
  ASSETS 2019 Workshop on AI Fairness for People with Disabilities | October 2019
  
  Publication Video Project
- Fair Regression: Quantitative Definitions and Reduction-Based Algorithms
  
  Alekh Agarwal, Miro Dudík, Zhiwei Steven Wu
  
  36th International Conference on Machine Learning | September 2019
  
  PDF Github
- Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries
  
  Alexandra Olteanu, Carlos Castillo, Fernando Diaz, Emre Kiciman
  
  Frontiers in Big Data | July 2019, Vol 2
  
  DOI Publication
- Improving fairness in machine learning systems: What do industry practitioners need?
  
  Ken Holstein, Jennifer Wortman Vaughan, Hal Daumé III, Miro Dudík, Hanna Wallach
  
  2019 ACM CHI Conference on Human Factors in Computing Systems | June 2019
  
  Publication
- Understanding the Effect of Accuracy on Trust in Machine Learning Models
  
  Ming Ying, Jennifer Wortman Vaughan, Hanna Wallach
  
  2019 ACM CHI Conference on Human Factors in Computing Systems | May 2019
  
  Honorable Mention Award at CHI 2019
  
  Publication Project
- What’s in a Name? Reducing Bias in Bios without Access to Protected Attributes
  
  Alexey Romanov, Maria De-Arteaga, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alex Chouldechova, Sahin Geyik, krishnaram Kenthapadi, Anna Rumshisky, Adam Tauman Kalai
  
  NAACL 2019 | April 2019
  
  Best Paper Award, NAACL 2019
  
  Publication
- The Disparate Effects of Strategic Manipulation
  
  Lily Hu, Nicole Immorlica, Jennifer Wortman Vaughan
  
  2nd ACM Conference on Fairness, Accountability, and Transparency | January 2019
  
  Publication
- The Externalities of Exploration and How Data Diversity Helps Exploitation
  
  Manish Raghavan, Aleksandrs Slivkins, Jennifer Wortman Vaughan, Zhiwei Steven Wu
  
  31st Annual Conference on Learning Theory | February 2018
  
  Publication
- Understanding and Meeting Practitioner Needs When Measuring Representational Harms Caused by LLM-Based Systems
  
  Emma Harvey, Emily Sheng, Su Lin Blodgett, Alex Chouldechova, Jean Garcia-Gathright, Alexandra Olteanu, Hanna Wallach
  
  Findings of the Association for Computational Linguistics: ACL 2025 | July 2025
  
  Publication
- Dehumanizing Machines: Mitigating Anthropomorphic Behaviors in Text Generation Systems
  
  Myra Cheng, Su Lin Blodgett, Alicia DeVrio, Lisa Egede, Alexandra Olteanu
  
  2025 Meeting of the Association for Computational Linguistics | July 2025
  
  SAC Highlight
  
  Publication
- Measuring Machine Learning Harms from Stereotypes: Requires Understanding Who is Being Harmed by Which Errors in What Ways
  
  Angelina Wang, Xuechunzi Bai, Solon Barocas, Su Lin Blodgett
  
  2025 ACM Conference on Fairness, Accountability, and Transparency | June 2025
  
  Publication
- “I Am the One and Only, Your Cyber BFF”: Understanding the Impact of GenAI Requires Understanding the Impact of Anthropomorphic AI
  
  Myra Cheng, Alicia DeVrio, Lisa Egede, Su Lin Blodgett, Alexandra Olteanu
  
  April 2025
  
  Publication Publication
- AI Automatons: AI Systems Intended to Imitate Humans
  
  Alexandra Olteanu, Solon Barocas, Su Lin Blodgett, Lisa Egede, Alicia DeVrio, Myra Cheng
  
  March 2025
  
  Preprint
- Metrics for What, Metrics for Whom: Assessing Actionability of Bias Evaluation Metrics in NLP
  
  Pieter Delobelle, Giuseppe Attanasio, Debora Nozza, Su Lin Blodgett, Zeerak Talat
  
  2024 Empirical Methods in Natural Language Processing | November 2024
  
  Publication
- The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels
  
  Eve Fleisig, Su Lin Blodgett, Dan Klein, Zeerak Talat
  
  NAACL 2024 | June 2024
  
  Publication
- “One-Size-Fits-All”? Examining Expectations around What Constitute “Fair” or “Good” NLG System Behaviors
  
  Li Lucy, Su Lin Blodgett, Milad Shokouhi, Hanna Wallach, Alexandra Olteanu
  
  NAACL 2024 | June 2024
  
  Publication
- Responsible AI Considerations in Text Summarization Research: A Review of Current Practices
  
  Yu Lu Liu, Meng Cao, Su Lin Blodgett, Jackie Chi Kit Cheung, Alexandra Olteanu, Adam Trischler
  
  Findings of EMNLP 2023 | December 2023
  
  Publication
- This Prompt is Measuring : Evaluating Bias Evaluation in Language Models
  
  Seraphina Goldfarb-Tarrant, Eddie Ungless, Esma Balkir, Su Lin Blodgett
  
  Findings of the Association for Computational Linguistics: ACL 2023 | July 2023
  
  Publication
- Deconstructing NLG Evaluation: Evaluation Practices, Assumptions, and Their Implications
  
  Kaitlyn Zhou, Su Lin Blodgett, Adam Trischler, Hal Daumé III, Kaheer Suleman, Alexandra Olteanu
  
  NAACL 2022 | May 2022
  
  Publication
- Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets
  
  Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, Hanna Wallach
  
  ACL-IJCNLP 2021 | August 2021
  
  Publication
- “I Can’t Reply with That”: Characterizing Problematic Email Reply Suggestions
  
  Ronald Robertson, Alexandra Olteanu, Fernando Diaz, Milad Shokouhi, Peter Bailey
  
  CHI Conference on Human Factors in Computing Systems (CHI ’21) | May 2021
  
  Publication Video
- Quand les suggestions de fin de recherche sont-elles problématiques ?
  
  Alexandra Olteanu, Fernando Diaz, Gabriella Kazai
  
  Computer Supported Collaborative Work and Social Computing (CSCW) | August 2020
  
  CSCW 2020 Honorable Mention
  
  CSCW 2020 Honorable Mention
  
  Publication Video
- On the Social and Technical Challenges of Web Search Autosuggestion Moderation
  
  T. J. Hazen, Alexandra Olteanu, Gabriella Kazai, Fernando Diaz, Michael Golebiewski
  
  July 2020
  
  Publication
- Language (Technology) is Power: A Critical Survey of “Bias” in NLP
  
  Su Lin Blodgett, Solon Barocas, Hal Daumé III, Hanna Wallach
  
  ACL | June 2020
  
  Publication Video
- Towards Query Logs For Privacy Studies: On Deriving Search Queries From Questions
  
  Asia J. Biega, Jana Schmidt, Rishiraj Saha Roy
  
  ECIR 2020 | April 2020
  
  Publication
- Overview of the TREC 2019 Fair Ranking Track
  
  Asia J. Biega, Fernando Diaz, Michael D. Ekstrand, Sebastian Kohlmeier
  
  TREC 2019 | November 2019
  
  Publication Publication
- What’s in a Name? Reducing Bias in Bios without Access to Protected Attributes
  
  Alexey Romanov, Maria De-Arteaga, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alex Chouldechova, Sahin Geyik, krishnaram Kenthapadi, Anna Rumshisky, Adam Tauman Kalai
  
  NAACL 2019 | April 2019
  
  Best Paper Award, NAACL 2019
  
  Publication
- Understanding and Meeting Practitioner Needs When Measuring Representational Harms Caused by LLM-Based Systems
  
  Emma Harvey, Emily Sheng, Su Lin Blodgett, Alex Chouldechova, Jean Garcia-Gathright, Alexandra Olteanu, Hanna Wallach
  
  Findings of the Association for Computational Linguistics: ACL 2025 | July 2025
  
  Publication
- Dehumanizing Machines: Mitigating Anthropomorphic Behaviors in Text Generation Systems
  
  Myra Cheng, Su Lin Blodgett, Alicia DeVrio, Lisa Egede, Alexandra Olteanu
  
  2025 Meeting of the Association for Computational Linguistics | July 2025
  
  SAC Highlight
  
  Publication
- Measuring Machine Learning Harms from Stereotypes: Requires Understanding Who is Being Harmed by Which Errors in What Ways
  
  Angelina Wang, Xuechunzi Bai, Solon Barocas, Su Lin Blodgett
  
  2025 ACM Conference on Fairness, Accountability, and Transparency | June 2025
  
  Publication
- Rigor in AI: Doing Rigorous AI Work Requires a Broader, Responsible AI-Informed Conception of Rigor
  
  Alexandra Olteanu, Su Lin Blodgett, Agathe Balayn, Angelina Wang, Fernando Diaz, Flavio du Pin Calmon, Margaret Mitchell, Michael Ekstrand, Reuben Binns, Solon Barocas
  
  June 2025
  
  Publication
- “I Am the One and Only, Your Cyber BFF”: Understanding the Impact of GenAI Requires Understanding the Impact of Anthropomorphic AI
  
  Myra Cheng, Alicia DeVrio, Lisa Egede, Su Lin Blodgett, Alexandra Olteanu
  
  April 2025
  
  Publication Publication
- Taxonomizing Representational Harms using Speech Act Theory
  
  Emily Corvi, Hannah Washington, Stefanie Reed, Chad Atalla, Alex Chouldechova, Alex Dow, Jean Garcia-Gathright, Nick Pangakis, Emily Sheng, Dan Vann, Matthew Vogel, Hanna Wallach
  
  March 2025
  
  Publication
- AI Automatons: AI Systems Intended to Imitate Humans
  
  Alexandra Olteanu, Solon Barocas, Su Lin Blodgett, Lisa Egede, Alicia DeVrio, Myra Cheng
  
  March 2025
  
  Preprint
- Position: Evaluating Generative AI Systems is a Social Science Measurement Challenge
  
  Hanna Wallach, Meera Desai, A. Feder Cooper, Angelina Wang, Chad Atalla, Solon Barocas, Su Lin Blodgett, Alex Chouldechova, Emily Corvi, Alex Dow, Jean Garcia-Gathright, Alexandra Olteanu, Nick Pangakis, Stefanie Reed, Emily Sheng, Dan Vann, Jennifer Wortman Vaughan, Matthew Vogel, Hannah Washington, Abigail Z. Jacobs
  
  ICML 2025 | January 2025
  
  Publication
- A Shared Standard for Valid Measurement of Generative AI Systems’ Capabilities, Risks, and Impacts
  
  Alex Chouldechova, Chad Atalla, Solon Barocas, A. Feder Cooper, Emily Corvi, Alex Dow, Jean Garcia-Gathright, Nick Pangakis, Stefanie Reed, Emily Sheng, Dan Vann, Matthew Vogel, Hannah Washington, Hanna Wallach
  
  December 2024
  
  Publication
- Gaps Between Research and Practice When Measuring Representational Harms Caused by LLM-Based Systems
  
  Emma Harvey, Emily Sheng, Su Lin Blodgett, Alex Chouldechova, Jean Garcia-Gathright, Alexandra Olteanu, Hanna Wallach
  
  November 2024
  
  Publication
- Dimensions of Generative AI Evaluation Design
  
  Alex Dow, Jennifer Wortman Vaughan, Solon Barocas, Chad Atalla, Alex Chouldechova, Hanna Wallach
  
  November 2024
  
  Publication
- Evaluating Generative AI Systems is a Social Science Measurement Challenge
  
  Hanna Wallach, Meera Desai, Nick Pangakis, A. Feder Cooper, Angelina Wang, Solon Barocas, Alex Chouldechova, Chad Atalla, Su Lin Blodgett, Emily Corvi, Alex Dow, Jean Garcia-Gathright, Alexandra Olteanu, Stefanie Reed, Emily Sheng, Dan Vann, Jennifer Wortman Vaughan, Matthew Vogel, Hannah Washington, Abigail Z. Jacobs
  
  November 2024
  
  Publication
- Un cadre pour explorer les conséquences de l'accès aux connaissances d'entreprise via l'IA et identifier les risques pour les travailleurs
  
  Anna Gausen, Bhaskar Mitra, Siân Lindley
  
  ACM Conference on Fairness, Accountability, and Transparency | March 2024
  
  Publication Preprint PDF Project Project
- Through the Looking-Glass: Transparency Implications and Challenges in Enterprise AI Knowledge Systems
  
  Karina Cortiñas-Lorenzo, Siân Lindley, Ida Larsen-Ledet, Bhaskar Mitra
  
  January 2024
  
  Preprint Project Project
- A Framework for Automated Measurement of Responsible AI Harms in Generative AI Applications
  
  Ahmed Magooda, Alec Helyar, Kyle Jackson, David Sullivan, Chad Atalla, Emily Sheng, Dan Vann, Richard Edgar, Hamid Palangi, Roman Lutz, Hongliang Kong, Vincent Yun, Eslam Kamal, Federico Zarfati, Hanna Wallach, Sarah Bird, Mei Chen
  
  ArXiv | October 2023, Vol abs/2310.17750
  
  DOI Publication Publication
- Responsible computing during COVID-19 and beyond
  
  Solon Barocas, Asia J. Biega, Margarita Boyarskaya, Kate Crawford, Hal Daumé III, Miro Dudík, Benjamin Fish, Mary L. Gray, Brent Hecht, Alexandra Olteanu, Forough Poursabzi-Sangdeh, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach, Marion Zepf
  
  Communications of the ACM | July 2021, Vol 67(7): pp. 30-32
  
  https://doi.org/10.1145/3466612
  
  Publication
- Measuring algorithmically infused societies
  
  Claudia Wagner, Markus Strohmaier, Alexandra Olteanu, Emre Kiciman, Noshir Contractor, Tina Eliassi-Rad
  
  Nature | June 2021, Vol 595: pp. 197-204
  
  Publication Publication
- Manipulating and Measuring Model Interpretability
  
  Forough Poursabzi-Sangdeh, Daniel G. Goldstein, Jake Hofman, Jennifer Wortman Vaughan, Hanna Wallach
  
  CHI 2021 | May 2021
  
  Publication Project
- Measurement and Fairness
  
  Abigail Z. Jacobs, Hanna Wallach
  
  Fairness, Accountability, and Transparency (FAccT ’21) | March 2021
  
  Publication DOI
- Overcoming Failures of Imagination in AI Infused System Development and Deployment
  
  Margarita Boyarskaya, Alexandra Olteanu, Kate Crawford
  
  In the Navigating the Broader Impacts of AI Research Workshop at NeurIPS 2020 | December 2020
  
  Publication Video
- Operationalizing the Legal Principle of Data Minimization for Personalization
  
  Asia J. Biega, Peter Potash, Hal Daumé III, Fernando Diaz, Michèle Finck
  
  SIGIR 2020 | July 2020
  
  PDF
- Language (Technology) is Power: A Critical Survey of “Bias” in NLP
  
  Su Lin Blodgett, Solon Barocas, Hal Daumé III, Hanna Wallach
  
  ACL | June 2020
  
  Publication Video
- When not to design, build, or deploy
  
  Solon Barocas, Asia J. Biega, Benjamin Fish, Jędrzej Niklas, Luke Stark
  
  2020 Conference on Fairness, Accountability, and Transparency (FAT* ’20) | January 2020
  
  Publication Author's Version
- Overview of the TREC 2019 Fair Ranking Track
  
  Asia J. Biega, Fernando Diaz, Michael D. Ekstrand, Sebastian Kohlmeier
  
  TREC 2019 | November 2019
  
  Publication Publication
- What’s in a Name? Reducing Bias in Bios without Access to Protected Attributes
  
  Alexey Romanov, Maria De-Arteaga, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alex Chouldechova, Sahin Geyik, krishnaram Kenthapadi, Anna Rumshisky, Adam Tauman Kalai
  
  NAACL 2019 | April 2019
  
  Best Paper Award, NAACL 2019
  
  Publication
- Position: To Make Text-to-Image Models that Work for Marginalized Communities, We Need New Measurement Practices for the Long Tail
  
  Nari Johnson, Hamna ., Deepthi Sudharsan, Theo Holroyd, Samantha Dalal, Siobhan Mackenzie Hall, Jennifer Wortman Vaughan, Daniela Massiceti, Cecily Morrison
  
  July 2025
  
  Preprint Project
- Supporting Industry Computing Researchers in Assessing, Articulating, and Addressing the Potential Negative Societal Impact of Their Work
  
  Wesley Hanwen Deng, Solon Barocas, Jennifer Wortman Vaughan
  
  ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2025) | November 2025
  
  Preprint
- Position: To Make Text-to-Image Models that Work for Marginalized Communities, We Need New Measurement Practices for the Long Tail
  
  Nari Johnson, Hamna ., Deepthi Sudharsan, Theo Holroyd, Samantha Dalal, Siobhan Mackenzie Hall, Jennifer Wortman Vaughan, Daniela Massiceti, Cecily Morrison
  
  July 2025
  
  Preprint Project
- Canvil: Designerly Adaptation for LLM-Powered User Experiences
  
  K. Feng, Q. Vera Liao, Ziang Xiao, Jennifer Wortman Vaughan, Amy Zhang, David W. McDonald
  
  CHI 2025 | April 2025
  
  DOI
- Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistencies
  
  Sunnie S. Y. Kim, Jennifer Wortman Vaughan, Q. Vera Liao, Tania Lombrozo, Olga Russakovsky
  
  CHI 2025 | April 2025
  
  Honorable Mention, CHI 2025
  
  Publication
- A Taxonomy of Linguistic Expressions That Contribute To Anthropomorphism of Language Technologies
  
  Alicia DeVrio, Myra Cheng, Lisa Egede, Alexandra Olteanu, Su Lin Blodgett
  
  CHI 2025 | April 2025
  
  Publication
- Generation Probabilities Are Not Enough: Uncertainty Highlighting in AI Code Completions
  
  Helena Vasconcelos, Gagan Bansal, Adam Fourney, Q. Vera Liao, Jennifer Wortman Vaughan
  
  ToCHI | April 2025, Vol 32(1)
  
  Publication
- Challenges in Human-Agent Communication
  
  Gagan Bansal, Jennifer Wortman Vaughan, Saleema Amershi, Eric Horvitz, Adam Fourney, Hussein Mozannar, Victor Dibia, Daniel S. Weld
  
  MSR-TR-2024-53 | December 2024
  
  Published by Microsoft
  
  Publication Project
- (De)Noise: Moderating the Inconsistency Between Human Decision-Makers
  
  Nina Grgić-Hlača, Junaid Ali, Krishna P. Gummadi, Jennifer Wortman Vaughan
  
  ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2024) | November 2024
  
  Publication
- Un cadre pour explorer les conséquences de l'accès aux connaissances d'entreprise via l'IA et identifier les risques pour les travailleurs
  
  Anna Gausen, Bhaskar Mitra, Siân Lindley
  
  ACM Conference on Fairness, Accountability, and Transparency | March 2024
  
  Publication Preprint PDF Project Project
- AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap
  
  Q. Vera Liao, Jennifer Wortman Vaughan
  
  Harvard Data Science Review | February 2024
  
  Publication Project
- Through the Looking-Glass: Transparency Implications and Challenges in Enterprise AI Knowledge Systems
  
  Karina Cortiñas-Lorenzo, Siân Lindley, Ida Larsen-Ledet, Bhaskar Mitra
  
  January 2024
  
  Preprint Project Project
- Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations
  
  Valerie Chen, Q. Vera Liao, Jennifer Wortman Vaughan, Gagan Bansal
  
  Proc. ACM Hum.-Comput. Interact. | October 2023, Vol 7
  
  Article 370
  
  PDF Project
- Patterns of gender-specializing query reformulation
  
  Amifa Raj, Bhaskar Mitra, Nick Craswell, Michael D. Ekstrand
  
  2023 International ACM SIGIR Conference on Research and Development in Information Retrieval | July 2023
  
  Publication Preprint PDF PDF
- fAIlureNotes: Supporting Designers in Understanding the Limits of AI Models for Computer Vision Tasks
  
  Steven Moore, Q. Vera Liao, Hariharan Subramonyam
  
  CHI 2023 | April 2023
  
  Preprint
- Designerly Understanding: Information Needs for Model Transparency to Support Design Ideation for AI-Powered User Experience
  
  Q. Vera Liao, Hariharan Subramonyam, Jennifer Wang, Jennifer Wortman Vaughan
  
  2023 ACM CHI Conference on Human Factors in Computing Systems (CHI 2023) | April 2023
  
  PDF Project
- GAM Coach: Towards Interactive and User-Centered Algorithmic Recourse
  
  Zijie J. Wang, Jennifer Wortman Vaughan, Rich Caruana, Duen Horng Chau
  
  2023 ACM CHI Conference on Human Factors in Computing Systems (CHI 2023) | February 2023
  
  PDF Project
- Understanding Machine Learning Practitioners’ Data Documentation Perceptions, Needs, Challenges, and Desiderata
  
  Amy K. Heger, Liz B. Marquis, Mihaela Vorvoreanu, Hanna Wallach, Jennifer Wortman Vaughan
  
  2022 Conference on Computer Supported Cooperative Work | November 2022
  
  Publication Project Project
- Generation Probabilities are Not Enough: Improving Error Highlighting for AI Code Suggestions
  
  Helena Vasconcelos, Gagan Bansal, Adam Fourney, Q. Vera Liao, Jennifer Wortman Vaughan
  
  NeurIPS Workshop on Human-Centered AI | October 2022
  
  Publication
- Connecting Algorithmic Research and Usage Contexts: A Perspective of Contextualized Evaluation for Explainable AI
  
  Q. Vera Liao, Yunfeng Zhang, Ronny Luss, Finale Doshi-Velez, Amit Dhurandhar
  
  AAAI Conference on Human Computation and Crowdsourcing | October 2022
  
  Best Paper Award, Honorable Mention
  
  Publication Project
- Ethical and Social Considerations in Automatic Expert Identification and People Recommendation in Organizational Knowledge Management Systems
  
  Ida Larsen-Ledet, Bhaskar Mitra, Siân Lindley
  
  Proceedings of the 5th FAccTRec Workshop on Responsible Recommendation at RecSys 2022 | September 2022
  
  Preprint PDF Project Project
- How Different Groups Prioritize Ethical Values for Responsible AI
  
  Maurice Jakesch, Zana Buçinca, Saleema Amershi, Alexandra Olteanu
  
  FAccT 2022 | May 2022
  
  Publication DOI
- Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation
  
  Vivian Lai, Samuel Carton, Rajat Bhatnagar, Q. Vera Liao, Yunfeng Zhang, Chenhao Tan
  
  CHI 2022 | May 2022
  
  Publication
- Designing for Responsible Trust in AI Systems: A Communication Perspective
  
  Q. Vera Liao, S. Shyam Sundar
  
  FAccT 2022 | April 2022
  
  Preprint
  
  Publication
- Responsible Language Technologies: Foreseeing and Mitigating Harms
  
  Su Lin Blodgett, Q. Vera Liao, Alexandra Olteanu, Rada Mihalcea, Michael Muller, Morgan Klaus Scheuerman, Chenhao Tan, Qian Yang
  
  CHI 2022 | April 2022
  
  Publication
- Responsible computing during COVID-19 and beyond
  
  Solon Barocas, Asia J. Biega, Margarita Boyarskaya, Kate Crawford, Hal Daumé III, Miro Dudík, Benjamin Fish, Mary L. Gray, Brent Hecht, Alexandra Olteanu, Forough Poursabzi-Sangdeh, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach, Marion Zepf
  
  Communications of the ACM | July 2021, Vol 67(7): pp. 30-32
  
  https://doi.org/10.1145/3466612
  
  Publication
- Measuring algorithmically infused societies
  
  Claudia Wagner, Markus Strohmaier, Alexandra Olteanu, Emre Kiciman, Noshir Contractor, Tina Eliassi-Rad
  
  Nature | June 2021, Vol 595: pp. 197-204
  
  Publication Publication
- “I Can’t Reply with That”: Characterizing Problematic Email Reply Suggestions
  
  Ronald Robertson, Alexandra Olteanu, Fernando Diaz, Milad Shokouhi, Peter Bailey
  
  CHI Conference on Human Factors in Computing Systems (CHI ’21) | May 2021
  
  Publication Video
- Quand les suggestions de fin de recherche sont-elles problématiques ?
  
  Alexandra Olteanu, Fernando Diaz, Gabriella Kazai
  
  Computer Supported Collaborative Work and Social Computing (CSCW) | August 2020
  
  CSCW 2020 Honorable Mention
  
  CSCW 2020 Honorable Mention
  
  Publication Video
- Interpreting Interpretability: Understanding Data Scientists’ Use of Interpretability Tools for Machine Learning
  
  Harmanpreet Kaur, Harsha Nori, Samuel Jenkins, Rich Caruana, Hanna Wallach, Jennifer Wortman Vaughan
  
  CHI 2020 | April 2020
  
  CHI 2020 Honorable Mention Award
  
  Publication Github Project
- Co-Designing Checklists to Understand Organizational Challenges and Opportunities around Fairness in AI
  
  Michael Madaio, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach
  
  CHI Conference on Human Factors in Computing Systems | March 2020
  
  CHI 2020 Best Paper Award
  
  PDF Project
- When not to design, build, or deploy
  
  Solon Barocas, Asia J. Biega, Benjamin Fish, Jędrzej Niklas, Luke Stark
  
  2020 Conference on Fairness, Accountability, and Transparency (FAT* ’20) | January 2020
  
  Publication Author's Version
- Toward Fairness in AI for People with Disabilities: A Research Roadmap
  
  Anhong Guo, Ece Kamar, Jennifer Wortman Vaughan, Hanna Wallach, Meredith Ringel Morris
  
  ASSETS 2019 Workshop on AI Fairness for People with Disabilities | October 2019
  
  Publication Video Project
- Improving fairness in machine learning systems: What do industry practitioners need?
  
  Ken Holstein, Jennifer Wortman Vaughan, Hal Daumé III, Miro Dudík, Hanna Wallach
  
  2019 ACM CHI Conference on Human Factors in Computing Systems | June 2019
  
  Publication
- Understanding the Effect of Accuracy on Trust in Machine Learning Models
  
  Ming Ying, Jennifer Wortman Vaughan, Hanna Wallach
  
  2019 ACM CHI Conference on Human Factors in Computing Systems | May 2019
  
  Honorable Mention Award at CHI 2019
  
  Publication Project
- The CLeAR Documentation Framework for AI Transparency: Recommendations for Practitioners & Context for Policymakers
  
  Kasia Chmielinski, Sarah Newman, Chris N. Kranzinger, Michael Hind, Jennifer Wortman Vaughan, Margaret Mitchell, Julia Stoyanovich, Angelina McMillan-Major, Emily McReynolds, Kathleen Esfahany, Mary L. Gray, Audrey Chang, Maui Hudson
  
  April 2024
  
  Publication Project
- Un cadre pour explorer les conséquences de l'accès aux connaissances d'entreprise via l'IA et identifier les risques pour les travailleurs
  
  Anna Gausen, Bhaskar Mitra, Siân Lindley
  
  ACM Conference on Fairness, Accountability, and Transparency | March 2024
  
  Publication Preprint PDF Project Project
- Through the Looking-Glass: Transparency Implications and Challenges in Enterprise AI Knowledge Systems
  
  Karina Cortiñas-Lorenzo, Siân Lindley, Ida Larsen-Ledet, Bhaskar Mitra
  
  January 2024
  
  Preprint Project Project
- Patterns of gender-specializing query reformulation
  
  Amifa Raj, Bhaskar Mitra, Nick Craswell, Michael D. Ekstrand
  
  2023 International ACM SIGIR Conference on Research and Development in Information Retrieval | July 2023
  
  Publication Preprint PDF PDF
- De-Biasing Relevance Judgements for Fair Ranking
  
  Amin Bigdeli, Negar Arabzadeh, Shirin Seyedsalehi, Bhaskar Mitra, Morteza Zihayat, Ebrahim Bagheri
  
  Proceedings of the 45th European Conference on Information Retrieval (ECIR) | January 2023
  
  Publication PDF PDF
- Ethical and Social Considerations in Automatic Expert Identification and People Recommendation in Organizational Knowledge Management Systems
  
  Ida Larsen-Ledet, Bhaskar Mitra, Siân Lindley
  
  Proceedings of the 5th FAccTRec Workshop on Responsible Recommendation at RecSys 2022 | September 2022
  
  Preprint PDF Project Project
- Joint Multisided Exposure Fairness for Recommendation
  
  Bhaskar Mitra, Haolun Wu, Chen Ma, Fernando Diaz, Xue Liu
  
  2022 International ACM SIGIR Conference on Research and Development in Information Retrieval | April 2022
  
  Publication PDF PDF
- Bias-aware Fair Neural Ranking for Addressing Stereotypical Gender Biases
  
  Shirin SeyedSalehi, Amin Bigdeli, Negar Arabzadeh, Bhaskar Mitra, Morteza Zihayat, Ebrahim Bagheri
  
  Proceedings of the 25th International Conference on Extending Database Technology (EDBT) | February 2022
  
  Publication PDF PDF
- Exposing Query Identification for Search Transparency
  
  Ruohan Li, Jianxiang Li, Bhaskar Mitra, Fernando Diaz, Asia Biega
  
  2022 The Web Conference | January 2022
  
  Publication Preprint PDF PDF
- Revisiting Popularity and Demographic Biases in Recommender Evaluation and Effectiveness
  
  Nicola Neophytou, Bhaskar Mitra, Catherine Stinson
  
  European Conference on Information Retrieval | November 2021
  
  Publication Preprint PDF PDF
- Quand les suggestions de fin de recherche sont-elles problématiques ?
  
  Alexandra Olteanu, Fernando Diaz, Gabriella Kazai
  
  Computer Supported Collaborative Work and Social Computing (CSCW) | August 2020
  
  CSCW 2020 Honorable Mention
  
  CSCW 2020 Honorable Mention
  
  Publication Video
- Operationalizing the Legal Principle of Data Minimization for Personalization
  
  Asia J. Biega, Peter Potash, Hal Daumé III, Fernando Diaz, Michèle Finck
  
  SIGIR 2020 | July 2020
  
  PDF
- On the Social and Technical Challenges of Web Search Autosuggestion Moderation
  
  T. J. Hazen, Alexandra Olteanu, Gabriella Kazai, Fernando Diaz, Michael Golebiewski
  
  July 2020
  
  Publication
- Evaluating Stochastic Rankings with Expected Exposure
  
  Fernando Diaz, Bhaskar Mitra, Michael D. Ekstrand, Asia J. Biega, Ben Carterette
  
  Proceedings of the 29th ACM International Conference on Information and Knowledge Management (CIKM) | July 2020
  
  Best Long Research Paper Nominee
  
  Publication Preprint PDF
- Towards Query Logs For Privacy Studies: On Deriving Search Queries From Questions
  
  Asia J. Biega, Jana Schmidt, Rishiraj Saha Roy
  
  ECIR 2020 | April 2020
  
  Publication
- FACTS-IR: Fairness, Accountability, Confidentiality, Transparency, and Safety in Information Retrieval
  
  Alexandra Olteanu, Jean Garcia-Gathright, Maarten de Rijke, Michael D. Ekstrand
  
  ACM SIGIR Forum | December 2019, Vol 53(2): pp. 20
  
  Publication
- Overview of the TREC 2019 Fair Ranking Track
  
  Asia J. Biega, Fernando Diaz, Michael D. Ekstrand, Sebastian Kohlmeier
  
  TREC 2019 | November 2019
  
  Publication Publication
- Overcoming Failures of Imagination in AI Infused System Development and Deployment
  
  Margarita Boyarskaya, Alexandra Olteanu, Kate Crawford
  
  In the Navigating the Broader Impacts of AI Research Workshop at NeurIPS 2020 | December 2020
  
  Publication Video
- Roles for computing in social change
  
  Rediet Abebe, Solon Barocas, Jon Kleinberg, Karen Levy, Manish Raghavan, David Robinson
  
  FAT* | January 2020
  
  Publication
- The Externalities of Exploration and How Data Diversity Helps Exploitation
  
  Manish Raghavan, Aleksandrs Slivkins, Jennifer Wortman Vaughan, Zhiwei Steven Wu
  
  31st Annual Conference on Learning Theory | February 2018
  
  Publication
- Operationalizing the Legal Principle of Data Minimization for Personalization
  
  Asia J. Biega, Peter Potash, Hal Daumé III, Fernando Diaz, Michèle Finck
  
  SIGIR 2020 | July 2020
  
  PDF
- Towards Query Logs For Privacy Studies: On Deriving Search Queries From Questions
  
  Asia J. Biega, Jana Schmidt, Rishiraj Saha Roy
  
  ECIR 2020 | April 2020
  
  Publication
- “I Don’t Want Someone to Watch Me While I’m Working”: Gendered Views of Facial Recognition Technology in Workplace Surveillance
  
  Luke Stark, Amanda Stanhaus, Denise L. Anthony
  
  Journal of the Association for Information Science and Technology | March 2020
  
  Publication
- Toward Fairness in AI for People with Disabilities: A Research Roadmap
  
  Anhong Guo, Ece Kamar, Jennifer Wortman Vaughan, Hanna Wallach, Meredith Ringel Morris
  
  ASSETS 2019 Workshop on AI Fairness for People with Disabilities | October 2019
  
  Publication Video Project
- The Disparate Effects of Strategic Manipulation
  
  Lily Hu, Nicole Immorlica, Jennifer Wortman Vaughan
  
  2nd ACM Conference on Fairness, Accountability, and Transparency | January 2019
  
  Publication
- The Externalities of Exploration and How Data Diversity Helps Exploitation
  
  Manish Raghavan, Aleksandrs Slivkins, Jennifer Wortman Vaughan, Zhiwei Steven Wu
  
  31st Annual Conference on Learning Theory | February 2018
  
  Publication

- “I Am the One and Only, Your Cyber BFF”: Understanding the Impact of GenAI Requires Understanding the Impact of Anthropomorphic AI
  
  Myra Cheng, Alicia DeVrio, Lisa Egede, Su Lin Blodgett, Alexandra Olteanu
  
  April 2025
  
  Publication Publication
- Taxonomizing Representational Harms using Speech Act Theory
  
  Emily Corvi, Hannah Washington, Stefanie Reed, Chad Atalla, Alex Chouldechova, Alex Dow, Jean Garcia-Gathright, Nick Pangakis, Emily Sheng, Dan Vann, Matthew Vogel, Hanna Wallach
  
  March 2025
  
  Publication
- A Shared Standard for Valid Measurement of Generative AI Systems’ Capabilities, Risks, and Impacts
  
  Alex Chouldechova, Chad Atalla, Solon Barocas, A. Feder Cooper, Emily Corvi, Alex Dow, Jean Garcia-Gathright, Nick Pangakis, Stefanie Reed, Emily Sheng, Dan Vann, Matthew Vogel, Hannah Washington, Hanna Wallach
  
  December 2024
  
  Publication
- Gaps Between Research and Practice When Measuring Representational Harms Caused by LLM-Based Systems
  
  Emma Harvey, Emily Sheng, Su Lin Blodgett, Alex Chouldechova, Jean Garcia-Gathright, Alexandra Olteanu, Hanna Wallach
  
  November 2024
  
  Publication
- Dimensions of Generative AI Evaluation Design
  
  Alex Dow, Jennifer Wortman Vaughan, Solon Barocas, Chad Atalla, Alex Chouldechova, Hanna Wallach
  
  November 2024
  
  Publication
- Evaluating Generative AI Systems is a Social Science Measurement Challenge
  
  Hanna Wallach, Meera Desai, Nick Pangakis, A. Feder Cooper, Angelina Wang, Solon Barocas, Alex Chouldechova, Chad Atalla, Su Lin Blodgett, Emily Corvi, Alex Dow, Jean Garcia-Gathright, Alexandra Olteanu, Stefanie Reed, Emily Sheng, Dan Vann, Jennifer Wortman Vaughan, Matthew Vogel, Hannah Washington, Abigail Z. Jacobs
  
  November 2024
  
  Publication
- The CLeAR Documentation Framework for AI Transparency: Recommendations for Practitioners & Context for Policymakers
  
  Kasia Chmielinski, Sarah Newman, Chris N. Kranzinger, Michael Hind, Jennifer Wortman Vaughan, Margaret Mitchell, Julia Stoyanovich, Angelina McMillan-Major, Emily McReynolds, Kathleen Esfahany, Mary L. Gray, Audrey Chang, Maui Hudson
  
  April 2024
  
  Publication Project
- A Human in the Loop is Not Enough: The Need for Human-Subject Experiments in Facial Recognition
  
  Forough Poursabzi-Sangdeh, Samira Samadi, Jennifer Wortman Vaughan, Hanna Wallach
  
  April 2020
  
  PDF
- Supporting Industry Computing Researchers in Assessing, Articulating, and Addressing the Potential Negative Societal Impact of Their Work
  
  Wesley Hanwen Deng, Solon Barocas, Jennifer Wortman Vaughan
  
  ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2025) | November 2025
  
  Preprint
- Understanding and Meeting Practitioner Needs When Measuring Representational Harms Caused by LLM-Based Systems
  
  Emma Harvey, Emily Sheng, Su Lin Blodgett, Alex Chouldechova, Jean Garcia-Gathright, Alexandra Olteanu, Hanna Wallach
  
  Findings of the Association for Computational Linguistics: ACL 2025 | July 2025
  
  Publication
- Dehumanizing Machines: Mitigating Anthropomorphic Behaviors in Text Generation Systems
  
  Myra Cheng, Su Lin Blodgett, Alicia DeVrio, Lisa Egede, Alexandra Olteanu
  
  2025 Meeting of the Association for Computational Linguistics | July 2025
  
  SAC Highlight
  
  Publication
- Measuring Machine Learning Harms from Stereotypes: Requires Understanding Who is Being Harmed by Which Errors in What Ways
  
  Angelina Wang, Xuechunzi Bai, Solon Barocas, Su Lin Blodgett
  
  2025 ACM Conference on Fairness, Accountability, and Transparency | June 2025
  
  Publication
- Examining the Expanding Role of Synthetic Data Throughout the AI Development Pipeline
  
  Shivani Kapania, Stephanie Ballard, Alex Kessler, Jennifer Wortman Vaughan
  
  8th ACM Conference on Fairness, Accountability, and Transparency (FAccT 2025) | June 2025
  
  Preprint
- Canvil: Designerly Adaptation for LLM-Powered User Experiences
  
  K. Feng, Q. Vera Liao, Ziang Xiao, Jennifer Wortman Vaughan, Amy Zhang, David W. McDonald
  
  CHI 2025 | April 2025
  
  DOI
- Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistencies
  
  Sunnie S. Y. Kim, Jennifer Wortman Vaughan, Q. Vera Liao, Tania Lombrozo, Olga Russakovsky
  
  CHI 2025 | April 2025
  
  Honorable Mention, CHI 2025
  
  Publication
- A Taxonomy of Linguistic Expressions That Contribute To Anthropomorphism of Language Technologies
  
  Alicia DeVrio, Myra Cheng, Lisa Egede, Alexandra Olteanu, Su Lin Blodgett
  
  CHI 2025 | April 2025
  
  Publication
- Position: Evaluating Generative AI Systems is a Social Science Measurement Challenge
  
  Hanna Wallach, Meera Desai, A. Feder Cooper, Angelina Wang, Chad Atalla, Solon Barocas, Su Lin Blodgett, Alex Chouldechova, Emily Corvi, Alex Dow, Jean Garcia-Gathright, Alexandra Olteanu, Nick Pangakis, Stefanie Reed, Emily Sheng, Dan Vann, Jennifer Wortman Vaughan, Matthew Vogel, Hannah Washington, Abigail Z. Jacobs
  
  ICML 2025 | January 2025
  
  Publication
- (De)Noise: Moderating the Inconsistency Between Human Decision-Makers
  
  Nina Grgić-Hlača, Junaid Ali, Krishna P. Gummadi, Jennifer Wortman Vaughan
  
  ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2024) | November 2024
  
  Publication
- Metrics for What, Metrics for Whom: Assessing Actionability of Bias Evaluation Metrics in NLP
  
  Pieter Delobelle, Giuseppe Attanasio, Debora Nozza, Su Lin Blodgett, Zeerak Talat
  
  2024 Empirical Methods in Natural Language Processing | November 2024
  
  Publication
- The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels
  
  Eve Fleisig, Su Lin Blodgett, Dan Klein, Zeerak Talat
  
  NAACL 2024 | June 2024
  
  Publication
- “One-Size-Fits-All”? Examining Expectations around What Constitute “Fair” or “Good” NLG System Behaviors
  
  Li Lucy, Su Lin Blodgett, Milad Shokouhi, Hanna Wallach, Alexandra Olteanu
  
  NAACL 2024 | June 2024
  
  Publication
- “I’m Not Sure, But…”: Examining the Impact of Large Language Models’ Uncertainty Expression on User Reliance and Trust
  
  Sunnie S. Y. Kim, Q. Vera Liao, Mihaela Vorvoreanu, Stephanie Ballard, Jennifer Wortman Vaughan
  
  7th ACM Conference on Fairness, Accountability, and Transparency (FAccT 2024) | June 2024
  
  Publication Project
- Tinker, Tailor, Configure, Customize: The Articulation Work of Customizing AI Fairness Checklists
  
  Michael Madaio, Jingya Chen, Hanna Wallach, Jennifer Wortman Vaughan
  
  Computer-Supported Cooperative Work | April 2024
  
  Publication Project
- Un cadre pour explorer les conséquences de l'accès aux connaissances d'entreprise via l'IA et identifier les risques pour les travailleurs
  
  Anna Gausen, Bhaskar Mitra, Siân Lindley
  
  ACM Conference on Fairness, Accountability, and Transparency | March 2024
  
  Publication Preprint PDF Project Project
- Responsible AI Considerations in Text Summarization Research: A Review of Current Practices
  
  Yu Lu Liu, Meng Cao, Su Lin Blodgett, Jackie Chi Kit Cheung, Alexandra Olteanu, Adam Trischler
  
  Findings of EMNLP 2023 | December 2023
  
  Publication
- Patterns of gender-specializing query reformulation
  
  Amifa Raj, Bhaskar Mitra, Nick Craswell, Michael D. Ekstrand
  
  2023 International ACM SIGIR Conference on Research and Development in Information Retrieval | July 2023
  
  Publication Preprint PDF PDF
- FairPrism: Evaluating Fairness-Related Harms in Text Generation
  
  Eve Fleisig, Aubrie Amstutz, Chad Atalla, Su Lin Blodgett, Hal Daumé III, Alexandra Olteanu, Emily Sheng, Dan Vann, Hanna Wallach
  
  ACL 2023 | July 2023
  
  Publication
- This Prompt is Measuring : Evaluating Bias Evaluation in Language Models
  
  Seraphina Goldfarb-Tarrant, Eddie Ungless, Esma Balkir, Su Lin Blodgett
  
  Findings of the Association for Computational Linguistics: ACL 2023 | July 2023
  
  Publication
- fAIlureNotes: Supporting Designers in Understanding the Limits of AI Models for Computer Vision Tasks
  
  Steven Moore, Q. Vera Liao, Hariharan Subramonyam
  
  CHI 2023 | April 2023
  
  Preprint
- Designerly Understanding: Information Needs for Model Transparency to Support Design Ideation for AI-Powered User Experience
  
  Q. Vera Liao, Hariharan Subramonyam, Jennifer Wang, Jennifer Wortman Vaughan
  
  2023 ACM CHI Conference on Human Factors in Computing Systems (CHI 2023) | April 2023
  
  PDF Project
- Taxonomizing and Measuring Representational Harms: A Look at Image Tagging
  
  Jared Katzman, Angelina Wang, Morgan Scheuerman, Su Lin Blodgett, Kristen Laird, Hanna Wallach, Solon Barocas
  
  AAAI 2022 | February 2023
  
  Publication
- GAM Coach: Towards Interactive and User-Centered Algorithmic Recourse
  
  Zijie J. Wang, Jennifer Wortman Vaughan, Rich Caruana, Duen Horng Chau
  
  2023 ACM CHI Conference on Human Factors in Computing Systems (CHI 2023) | February 2023
  
  PDF Project
- De-Biasing Relevance Judgements for Fair Ranking
  
  Amin Bigdeli, Negar Arabzadeh, Shirin Seyedsalehi, Bhaskar Mitra, Morteza Zihayat, Ebrahim Bagheri
  
  Proceedings of the 45th European Conference on Information Retrieval (ECIR) | January 2023
  
  Publication PDF PDF
- Understanding Machine Learning Practitioners’ Data Documentation Perceptions, Needs, Challenges, and Desiderata
  
  Amy K. Heger, Liz B. Marquis, Mihaela Vorvoreanu, Hanna Wallach, Jennifer Wortman Vaughan
  
  2022 Conference on Computer Supported Cooperative Work | November 2022
  
  Publication Project Project
- Generation Probabilities are Not Enough: Improving Error Highlighting for AI Code Suggestions
  
  Helena Vasconcelos, Gagan Bansal, Adam Fourney, Q. Vera Liao, Jennifer Wortman Vaughan
  
  NeurIPS Workshop on Human-Centered AI | October 2022
  
  Publication
- Connecting Algorithmic Research and Usage Contexts: A Perspective of Contextualized Evaluation for Explainable AI
  
  Q. Vera Liao, Yunfeng Zhang, Ronny Luss, Finale Doshi-Velez, Amit Dhurandhar
  
  AAAI Conference on Human Computation and Crowdsourcing | October 2022
  
  Best Paper Award, Honorable Mention
  
  Publication Project
- Ethical and Social Considerations in Automatic Expert Identification and People Recommendation in Organizational Knowledge Management Systems
  
  Ida Larsen-Ledet, Bhaskar Mitra, Siân Lindley
  
  Proceedings of the 5th FAccTRec Workshop on Responsible Recommendation at RecSys 2022 | September 2022
  
  Preprint PDF Project Project
- Examining Responsibility and Deliberation in AI Impact Statements and Ethics Reviews
  
  David Liu, Priyanka Nanayakkara, Sarah Ariyan Sakha, Grace Abuhamad, Su Lin Blodgett, Nicholas Diakopoulos, Jessica R. Hullman, Tina Eliassi-Rad
  
  AIES 2022 | July 2022
  
  Publication
- Interpretability, Then What? Editing Machine Learning Models to Reflect Human Knowledge and Values
  
  Zijie J. Wang, Alex Kale, Harsha Nori, Peter Stella, Mark E. Nunnally, Duen Horng Chau, Mihaela Vorvoreanu, Jennifer Wortman Vaughan, Rich Caruana
  
  KDD 2022 | July 2022
  
  Publication Project
- How Different Groups Prioritize Ethical Values for Responsible AI
  
  Maurice Jakesch, Zana Buçinca, Saleema Amershi, Alexandra Olteanu
  
  FAccT 2022 | May 2022
  
  Publication DOI
- Deconstructing NLG Evaluation: Evaluation Practices, Assumptions, and Their Implications
  
  Kaitlyn Zhou, Su Lin Blodgett, Adam Trischler, Hal Daumé III, Kaheer Suleman, Alexandra Olteanu
  
  NAACL 2022 | May 2022
  
  Publication
- REAL ML: Recognizing, Exploring, and Articulating Limitations of Machine Learning Research
  
  Jessie J. Smith, Saleema Amershi, Solon Barocas, Hanna Wallach, Jennifer Wortman Vaughan
  
  FAccT 2022 | May 2022
  
  DOI Project
- Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation
  
  Vivian Lai, Samuel Carton, Rajat Bhatnagar, Q. Vera Liao, Yunfeng Zhang, Chenhao Tan
  
  CHI 2022 | May 2022
  
  Publication
- Designing for Responsible Trust in AI Systems: A Communication Perspective
  
  Q. Vera Liao, S. Shyam Sundar
  
  FAccT 2022 | April 2022
  
  Preprint
  
  Publication
- Joint Multisided Exposure Fairness for Recommendation
  
  Bhaskar Mitra, Haolun Wu, Chen Ma, Fernando Diaz, Xue Liu
  
  2022 International ACM SIGIR Conference on Research and Development in Information Retrieval | April 2022
  
  Publication PDF PDF
- Responsible Language Technologies: Foreseeing and Mitigating Harms
  
  Su Lin Blodgett, Q. Vera Liao, Alexandra Olteanu, Rada Mihalcea, Michael Muller, Morgan Klaus Scheuerman, Chenhao Tan, Qian Yang
  
  CHI 2022 | April 2022
  
  Publication
- Assessing the Fairness of AI Systems: AI Practitioners’ Processes, Challenges, and Needs for Support
  
  Michael Madaio, Lisa Egede, Hariharan Subramonyam, Jennifer Wortman Vaughan, Hanna Wallach
  
  25th ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2022) | February 2022
  
  Publication
- Bias-aware Fair Neural Ranking for Addressing Stereotypical Gender Biases
  
  Shirin SeyedSalehi, Amin Bigdeli, Negar Arabzadeh, Bhaskar Mitra, Morteza Zihayat, Ebrahim Bagheri
  
  Proceedings of the 25th International Conference on Extending Database Technology (EDBT) | February 2022
  
  Publication PDF PDF
- Exposing Query Identification for Search Transparency
  
  Ruohan Li, Jianxiang Li, Bhaskar Mitra, Fernando Diaz, Asia Biega
  
  2022 The Web Conference | January 2022
  
  Publication Preprint PDF PDF
- Revisiting Popularity and Demographic Biases in Recommender Evaluation and Effectiveness
  
  Nicola Neophytou, Bhaskar Mitra, Catherine Stinson
  
  European Conference on Information Retrieval | November 2021
  
  Publication Preprint PDF PDF
- Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets
  
  Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, Hanna Wallach
  
  ACL-IJCNLP 2021 | August 2021
  
  Publication
- Manipulating and Measuring Model Interpretability
  
  Forough Poursabzi-Sangdeh, Daniel G. Goldstein, Jake Hofman, Jennifer Wortman Vaughan, Hanna Wallach
  
  CHI 2021 | May 2021
  
  Publication Project
- Designing Disaggregated Evaluations of AI Systems: Choices, Considerations, and Tradeoffs
  
  Solon Barocas, Anhong Guo, Ece Kamar, Jacquelyn Krones, Meredith Ringel Morris, Jennifer Wortman Vaughan, Duncan Wadsworth, Hanna Wallach
  
  AIES 2021 | May 2021
  
  Publication
- “I Can’t Reply with That”: Characterizing Problematic Email Reply Suggestions
  
  Ronald Robertson, Alexandra Olteanu, Fernando Diaz, Milad Shokouhi, Peter Bailey
  
  CHI Conference on Human Factors in Computing Systems (CHI ’21) | May 2021
  
  Publication Video
- Measurement and Fairness
  
  Abigail Z. Jacobs, Hanna Wallach
  
  Fairness, Accountability, and Transparency (FAccT ’21) | March 2021
  
  Publication DOI
- Overcoming Failures of Imagination in AI Infused System Development and Deployment
  
  Margarita Boyarskaya, Alexandra Olteanu, Kate Crawford
  
  In the Navigating the Broader Impacts of AI Research Workshop at NeurIPS 2020 | December 2020
  
  Publication Video
- Mitigating bias in algorithmic hiring: evaluating claims and practices
  
  Manish Raghavan, Solon Barocas, Jon Kleinberg, Karen Levy
  
  FAT* 2020 | August 2020
  
  Publication
- The Hidden Assumptions Behind Counterfactual Explanations and Principal Reasons
  
  Solon Barocas, Andrew Selbst, Manish Raghavan
  
  FAT* 2020 | August 2020
  
  Publication Project
- Quand les suggestions de fin de recherche sont-elles problématiques ?
  
  Alexandra Olteanu, Fernando Diaz, Gabriella Kazai
  
  Computer Supported Collaborative Work and Social Computing (CSCW) | August 2020
  
  CSCW 2020 Honorable Mention
  
  CSCW 2020 Honorable Mention
  
  Publication Video
- Operationalizing the Legal Principle of Data Minimization for Personalization
  
  Asia J. Biega, Peter Potash, Hal Daumé III, Fernando Diaz, Michèle Finck
  
  SIGIR 2020 | July 2020
  
  PDF
- Evaluating Stochastic Rankings with Expected Exposure
  
  Fernando Diaz, Bhaskar Mitra, Michael D. Ekstrand, Asia J. Biega, Ben Carterette
  
  Proceedings of the 29th ACM International Conference on Information and Knowledge Management (CIKM) | July 2020
  
  Best Long Research Paper Nominee
  
  Publication Preprint PDF
- Language (Technology) is Power: A Critical Survey of “Bias” in NLP
  
  Su Lin Blodgett, Solon Barocas, Hal Daumé III, Hanna Wallach
  
  ACL | June 2020
  
  Publication Video
- Towards Query Logs For Privacy Studies: On Deriving Search Queries From Questions
  
  Asia J. Biega, Jana Schmidt, Rishiraj Saha Roy
  
  ECIR 2020 | April 2020
  
  Publication
- Interpreting Interpretability: Understanding Data Scientists’ Use of Interpretability Tools for Machine Learning
  
  Harmanpreet Kaur, Harsha Nori, Samuel Jenkins, Rich Caruana, Hanna Wallach, Jennifer Wortman Vaughan
  
  CHI 2020 | April 2020
  
  CHI 2020 Honorable Mention Award
  
  Publication Github Project
- Co-Designing Checklists to Understand Organizational Challenges and Opportunities around Fairness in AI
  
  Michael Madaio, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach
  
  CHI Conference on Human Factors in Computing Systems | March 2020
  
  CHI 2020 Best Paper Award
  
  PDF Project
- Roles for computing in social change
  
  Rediet Abebe, Solon Barocas, Jon Kleinberg, Karen Levy, Manish Raghavan, David Robinson
  
  FAT* | January 2020
  
  Publication
- When not to design, build, or deploy
  
  Solon Barocas, Asia J. Biega, Benjamin Fish, Jędrzej Niklas, Luke Stark
  
  2020 Conference on Fairness, Accountability, and Transparency (FAT* ’20) | January 2020
  
  Publication Author's Version
- Overview of the TREC 2019 Fair Ranking Track
  
  Asia J. Biega, Fernando Diaz, Michael D. Ekstrand, Sebastian Kohlmeier
  
  TREC 2019 | November 2019
  
  Publication Publication
- Toward Fairness in AI for People with Disabilities: A Research Roadmap
  
  Anhong Guo, Ece Kamar, Jennifer Wortman Vaughan, Hanna Wallach, Meredith Ringel Morris
  
  ASSETS 2019 Workshop on AI Fairness for People with Disabilities | October 2019
  
  Publication Video Project
- Fair Regression: Quantitative Definitions and Reduction-Based Algorithms
  
  Alekh Agarwal, Miro Dudík, Zhiwei Steven Wu
  
  36th International Conference on Machine Learning | September 2019
  
  PDF Github
- Improving fairness in machine learning systems: What do industry practitioners need?
  
  Ken Holstein, Jennifer Wortman Vaughan, Hal Daumé III, Miro Dudík, Hanna Wallach
  
  2019 ACM CHI Conference on Human Factors in Computing Systems | June 2019
  
  Publication
- Understanding the Effect of Accuracy on Trust in Machine Learning Models
  
  Ming Ying, Jennifer Wortman Vaughan, Hanna Wallach
  
  2019 ACM CHI Conference on Human Factors in Computing Systems | May 2019
  
  Honorable Mention Award at CHI 2019
  
  Publication Project
- What’s in a Name? Reducing Bias in Bios without Access to Protected Attributes
  
  Alexey Romanov, Maria De-Arteaga, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alex Chouldechova, Sahin Geyik, krishnaram Kenthapadi, Anna Rumshisky, Adam Tauman Kalai
  
  NAACL 2019 | April 2019
  
  Best Paper Award, NAACL 2019
  
  Publication
- The Disparate Effects of Strategic Manipulation
  
  Lily Hu, Nicole Immorlica, Jennifer Wortman Vaughan
  
  2nd ACM Conference on Fairness, Accountability, and Transparency | January 2019
  
  Publication
- The Externalities of Exploration and How Data Diversity Helps Exploitation
  
  Manish Raghavan, Aleksandrs Slivkins, Jennifer Wortman Vaughan, Zhiwei Steven Wu
  
  31st Annual Conference on Learning Theory | February 2018
  
  Publication
- Position: To Make Text-to-Image Models that Work for Marginalized Communities, We Need New Measurement Practices for the Long Tail
  
  Nari Johnson, Hamna ., Deepthi Sudharsan, Theo Holroyd, Samantha Dalal, Siobhan Mackenzie Hall, Jennifer Wortman Vaughan, Daniela Massiceti, Cecily Morrison
  
  July 2025
  
  Preprint Project
- Rigor in AI: Doing Rigorous AI Work Requires a Broader, Responsible AI-Informed Conception of Rigor
  
  Alexandra Olteanu, Su Lin Blodgett, Agathe Balayn, Angelina Wang, Fernando Diaz, Flavio du Pin Calmon, Margaret Mitchell, Michael Ekstrand, Reuben Binns, Solon Barocas
  
  June 2025
  
  Publication
- AI Automatons: AI Systems Intended to Imitate Humans
  
  Alexandra Olteanu, Solon Barocas, Su Lin Blodgett, Lisa Egede, Alicia DeVrio, Myra Cheng
  
  March 2025
  
  Preprint
- Through the Looking-Glass: Transparency Implications and Challenges in Enterprise AI Knowledge Systems
  
  Karina Cortiñas-Lorenzo, Siân Lindley, Ida Larsen-Ledet, Bhaskar Mitra
  
  January 2024
  
  Preprint Project Project
- On the Social and Technical Challenges of Web Search Autosuggestion Moderation
  
  T. J. Hazen, Alexandra Olteanu, Gabriella Kazai, Fernando Diaz, Michael Golebiewski
  
  July 2020
  
  Publication
- Generation Probabilities Are Not Enough: Uncertainty Highlighting in AI Code Completions
  
  Helena Vasconcelos, Gagan Bansal, Adam Fourney, Q. Vera Liao, Jennifer Wortman Vaughan
  
  ToCHI | April 2025, Vol 32(1)
  
  Publication
- UX Matters: The Critical Role of UX in Responsible AI
  
  Q. Vera Liao, Mihaela Vorvoreanu, Hariharan Subramonyam, Lauren Wilcox
  
  ACM Interactions | July 2024, Vol 31: pp. 22-27
  
  DOI Publication
- AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap
  
  Q. Vera Liao, Jennifer Wortman Vaughan
  
  Harvard Data Science Review | February 2024
  
  Publication Project
- A Framework for Automated Measurement of Responsible AI Harms in Generative AI Applications
  
  Ahmed Magooda, Alec Helyar, Kyle Jackson, David Sullivan, Chad Atalla, Emily Sheng, Dan Vann, Richard Edgar, Hamid Palangi, Roman Lutz, Hongliang Kong, Vincent Yun, Eslam Kamal, Federico Zarfati, Hanna Wallach, Sarah Bird, Mei Chen
  
  ArXiv | October 2023, Vol abs/2310.17750
  
  DOI Publication Publication
- Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations
  
  Valerie Chen, Q. Vera Liao, Jennifer Wortman Vaughan, Gagan Bansal
  
  Proc. ACM Hum.-Comput. Interact. | October 2023, Vol 7
  
  Article 370
  
  PDF Project
- Summarize with Caution: Comparing Global Feature Attributions
  
  Alex Okeson, Rich Caruana, Nick Craswell, Kori Inkpen, Scott M. Lundberg, Harsha Nori, Hanna Wallach, Jennifer Wortman Vaughan
  
  Bulletin of the IEEE Computer Society Technical Committee on Data Engineering | December 2021
  
  Publication Project
- Datasheets for Datasets
  
  Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, Kate Crawford
  
  Communications of the ACM | December 2021, Vol 64(12): pp. 86-92
  
  Publication Project Project
- Responsible computing during COVID-19 and beyond
  
  Solon Barocas, Asia J. Biega, Margarita Boyarskaya, Kate Crawford, Hal Daumé III, Miro Dudík, Benjamin Fish, Mary L. Gray, Brent Hecht, Alexandra Olteanu, Forough Poursabzi-Sangdeh, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach, Marion Zepf
  
  Communications of the ACM | July 2021, Vol 67(7): pp. 30-32
  
  https://doi.org/10.1145/3466612
  
  Publication
- Measuring algorithmically infused societies
  
  Claudia Wagner, Markus Strohmaier, Alexandra Olteanu, Emre Kiciman, Noshir Contractor, Tina Eliassi-Rad
  
  Nature | June 2021, Vol 595: pp. 197-204
  
  Publication Publication
- Privacy Dependencies
  
  Solon Barocas, Karen Levy
  
  Washington Law Review | June 2020, Vol 95(2): pp. 555-615
  
  Publication
- “I Don’t Want Someone to Watch Me While I’m Working”: Gendered Views of Facial Recognition Technology in Workplace Surveillance
  
  Luke Stark, Amanda Stanhaus, Denise L. Anthony
  
  Journal of the Association for Information Science and Technology | March 2020
  
  Publication
- FACTS-IR: Fairness, Accountability, Confidentiality, Transparency, and Safety in Information Retrieval
  
  Alexandra Olteanu, Jean Garcia-Gathright, Maarten de Rijke, Michael D. Ekstrand
  
  ACM SIGIR Forum | December 2019, Vol 53(2): pp. 20
  
  Publication
- Stretching Human Laws to Apply to Machines: The Dangers of a ‘Colorblind’ Computer
  
  Zach Harned, Hanna Wallach
  
  Florida State University Law Review, Forthcoming | December 2019
  
  Publication
- Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries
  
  Alexandra Olteanu, Carlos Castillo, Fernando Diaz, Emre Kiciman
  
  Frontiers in Big Data | July 2019, Vol 2
  
  DOI Publication
- Challenges in Human-Agent Communication
  
  Gagan Bansal, Jennifer Wortman Vaughan, Saleema Amershi, Eric Horvitz, Adam Fourney, Hussein Mozannar, Victor Dibia, Daniel S. Weld
  
  MSR-TR-2024-53 | December 2024
  
  Published by Microsoft
  
  Publication Project
- Assessing and mitigating unfairness in credit models with the Fairlearn toolkit
  
  Miro Dudík, William Chen, Solon Barocas, Mario Inchiosa, Nick Lewins, Miruna Oprescu, Joy Qiao, Mehrnoosh Sameki, Mario Schlener, Jason Tuo, Hanna Wallach
  
  MSR-TR-2020-34 | September 2020
  
  Published by Microsoft
  
  Publication
- Fairlearn: A toolkit for assessing and improving fairness in AI
  
  Sarah Bird, Miro Dudík, Richard Edgar, Brandon Horn, Roman Lutz, Vanessa Milan, Mehrnoosh Sameki, Hanna Wallach, Kathleen Walker
  
  MSR-TR-2020-32 | May 2020
  
  Published by Microsoft
  
  Publication
- A Human-Centered Agenda for Intelligible Machine Learning
  
  Jennifer Wortman Vaughan, Hanna Wallach
  
  In Machines We Trust: Perspectives on Dependable AI | Published by MIT Press | 2021
  
  Author's Version Project