Xia Song

CVP, Turing MSAI

Publications

View by:

- POROver: Improving Safety and Reducing Overrefusal in Large Language Models with Overgeneration and Preference Optimization
  
  Batuhan K. Karaman, I. Zabir, Alon Benhaim, Vishrav Chaudhary, M. Sabuncu, Xia Song
  
  ICML 2025 | October 2024
  
  DOI Publication Publication
- Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
  
  Marah I Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Hassan Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Caio César Teodoro Mendes, Weizhu Chen, Vishrav Chaudhary, Parul Chopra, Allie Del Giorno, Gustavo de Rosa, Matthew Dixon, Ronen Eldan, Dan Iter, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Jamie Huynh, Mojan Javaheripi, Xin Jin, Piero Kauffmann, Nikos Karampatziakis, Dongwoo Kim, Mahmoud Khademi, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Chen Liang, Weishung Liu, Xihui (Eric) Lin, Zeqi Lin, Piyush Madan, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez-Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac, Corby Rosset, Sambudha Roy, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang, Hiteshi Sharma, Xia Song, Olatunji Ruwase, Xin Wang, Rachel Ward, Guanhua Wang, Philipp Witte, Michael Wyatt, Can Xu, Jiahang Xu, Weijian Xu, Sonali Yadav, Fan Yang, Ziyi Yang, Donghan Yu, Chengruidong Zhang, Cyril Zhang, Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yunan Zhang, Xiren Zhou
  
  MSR-TR-2024-12 | August 2024
  
  Published by Microsoft
  
  Publication Video Download Project
- WildFeedback: Aligning LLMs With In-situ User Interactions And Feedback
  
  Taiwei Shi, Zhuoer Wang, Longqi Yang, Ying-Chun Lin, Zexue He, Mengting Wan, Pei Zhou, Sujay Kumar Jauhar, Xiaofeng Xu, Xia Song, Jennifer Neville
  
  ArXiv | August 2024, Vol abs/2408.15549
  
  Publication DOI
- Interpretable User Satisfaction Estimation for Conversational Systems with Large Language Models
  
  Ying-Chun Lin, Jennifer Neville, Jack W. Stokes, Longqi Yang, Tara Safavi, Mengting Wan, Scott Counts, Siddharth Suri, Reid Andersen, Xiaofeng Xu, Deepak Gupta, Sujay Kumar Jauhar, Xia Song, Georg Buscher, Saurabh Tiwary, Brent Hecht, Jaime Teevan
  
  ACL | August 2024
  
  Publication Project
- Model-Generated Pretraining Signals Improves Zero-Shot Generalization of Text-to-Text Transformers
  
  Linyuan Gong, Chenyan Xiong, Xiaodong Liu, Payal Bajaj, Yiqing Xie, Alvin Cheung, Jianfeng Gao, Xia Song
  
  ACL 2023 | July 2023
  
  Publication
- Magneto: A Foundation Transformer
  
  Hongyu Wang, Shuming Ma, Shaohan Huang, Li Dong, Wenhui Wang, Zhiliang Peng, Yu Wu, Payal Bajaj, Saksham Singhal, Alon Benhaim, Barun Patra, Zhun Liu, Vishrav Chaudhary, Xia Song, Furu Wei
  
  ICML 2023 | June 2023
  
  Publication
- Language Is Not All You Need: Aligning Perception with Language Models
  
  Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, Qiang Liu, Kriti Aggarwal, Zewen Chi, Johan Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, Furu Wei
  
  NeurIPS 2023 | March 2023
  
  Publication
- A Length-Extrapolatable Transformer
  
  Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, Furu Wei
  
  ACL 2023 | December 2022
  
  Publication
- Beyond English-Centric Bitexts for Better Multilingual Language Representation Learning
  
  Barun Patra, Saksham Singhal, Shaohan Huang, Zewen Chi, Li Dong, Furu Wei, Vishrav Chaudhary, Xia Song
  
  ACL 2023 | October 2022
  
  Work in progress
  
  Publication
- On the Representation Collapse of Sparse Mixture of Experts
  
  Zewen Chi, Li Dong, Shaohan Huang, Damai Dai, Shuming Ma, Barun Patra, Saksham Singhal, Payal Bajaj, Xia Song, Furu Wei
  
  2022 Neural Information Processing Systems | October 2022
  
  Publication
- METRO: Efficient Denoising Pretraining of Large Scale Autoencoding Language Models with Model Generated Signals
  
  Payal Bajaj, Chenyan Xiong, Guolin Ke, Xiaodong Liu, Saurabh Tiwary, Tie-Yan Liu, Paul Bennett, Xia Song, Jianfeng Gao
  
  April 2022
  
  arXiv:2204.06644
  
  Publication
- Pretraining Text Encoders with Adversarial Mixture of Training Signal Generators
  
  Yu Meng, Chenyan Xiong, Payal Bajaj, Saurabh Tiwary, Paul Bennett, Jiawei Han, Xia Song
  
  ICLR 2022 | April 2022
  
  Publication Github
- COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining
  
  Yu Meng, Chenyan Xiong, Payal Bajaj, Saurabh Tiwary, Paul Bennett, Jiawei Han, Xia Song
  
  NeurIPS 2021 | December 2021
  
  Publication
- Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task.
  
  Jian Yang, Shuming Ma, Haoyang Huang, Dongdong Zhang, Li Dong, Shaohan Huang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
  
  WMT | November 2021
  
  Publication Publication Project
- Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training
  
  Bo Zheng, Li Dong, Shaohan Huang, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, Furu Wei
  
  EMNLP 2021 | September 2021
  
  Publication Project
- XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
  
  Zewen Chi, Shaohan Huang, Li Dong, Shuming Ma, Saksham Singhal, Payal Bajaj, Xia Song, Furu Wei
  
  ACL 2022 | June 2021
  
  Publication Project
- DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders.
  
  Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
  
  arXiv: Computation and Language | June 2021
  
  Publication Publication Publication Project
- DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders.
  
  Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
  
  June 2021
  
  Publication Publication Project
- Consistency Regularization for Cross-Lingual Fine-Tuning
  
  Bo Zheng, Li Dong, Shaohan Huang, Wenhui Wang, Zewen Chi, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, Furu Wei
  
  ACL-IJCNLP 2021 | June 2021
  
  PDF
- InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
  
  Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, Ming Zhou
  
  June 2021
  
  Publication Publication
- Language Scaling for Universal Suggested Replies Model
  
  Qianlan Ying, Payal Bajaj, Budhaditya Deb, Yu Yang, Wei Wang, Bojia Lin, Milad Shokouhi, Xia Song, Yang Yang, Daxin Jiang (姜大昕)
  
  NAACL 2021 Industrial Track | June 2021
  
  Publication
- Knowledge-Aware Language Model Pretraining
  
  Corby Rosset, Chenyan Xiong, Minh Phan, Xia Song, Paul Bennett, Saurabh Tiwary
  
  February 2021
  
  arXiv:2007.00655
  
  Publication
- XLM-T: Scaling up Multilingual Machine Translation with Pretrained Cross-lingual Transformer Encoders.
  
  Shuming Ma, Jian Yang, Haoyang Huang, Zewen Chi, Li Dong, Dongdong Zhang, Hany Hassan Awadalla, Alexandre Muzio, Akiko I. Eriguchi, Saksham Singhal, Xia Song, Arul Menezes, Furu Wei
  
  December 2020
  
  Publication Publication Publication Project
- Pushing the Limits of Narrow Precision Inferencing at Cloud Scale with Microsoft Floating Point
  
  Bita Darvish Rouhani, Daniel Lo, Ritchie Zhao, Ming Liu, Jeremy Fowers, Kalin Ovtcharov, Anna Vinogradsky, Sarah Massengill, Lita Yang, Ray Bittner, Alessandro Forin, Haishan Zhu, Taesik Na, Prerak Patel, Shuai Che, Lok Chand Koppaka, Xia Song, Subhojit Som, Kaustav Das, Saurabh Tiwary, Steve Reinhardt, Sitaram Lanka, Eric Chung, Doug Burger
  
  NeurIPS 2020 | November 2020
  
  Publication
- InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
  
  Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, Ming Zhou
  
  July 2020
  
  Publication Publication Github
- Towards Language Agnostic Universal Representations
  
  Armen Aghajanyan, Xia Song, Saurabh Tiwary
  
  September 2018
  
  Publication Project

- POROver: Improving Safety and Reducing Overrefusal in Large Language Models with Overgeneration and Preference Optimization
  
  Batuhan K. Karaman, I. Zabir, Alon Benhaim, Vishrav Chaudhary, M. Sabuncu, Xia Song
  
  ICML 2025 | October 2024
  
  DOI Publication Publication
- Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
  
  Marah I Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Hassan Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Caio César Teodoro Mendes, Weizhu Chen, Vishrav Chaudhary, Parul Chopra, Allie Del Giorno, Gustavo de Rosa, Matthew Dixon, Ronen Eldan, Dan Iter, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Jamie Huynh, Mojan Javaheripi, Xin Jin, Piero Kauffmann, Nikos Karampatziakis, Dongwoo Kim, Mahmoud Khademi, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Chen Liang, Weishung Liu, Xihui (Eric) Lin, Zeqi Lin, Piyush Madan, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez-Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac, Corby Rosset, Sambudha Roy, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang, Hiteshi Sharma, Xia Song, Olatunji Ruwase, Xin Wang, Rachel Ward, Guanhua Wang, Philipp Witte, Michael Wyatt, Can Xu, Jiahang Xu, Weijian Xu, Sonali Yadav, Fan Yang, Ziyi Yang, Donghan Yu, Chengruidong Zhang, Cyril Zhang, Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yunan Zhang, Xiren Zhou
  
  MSR-TR-2024-12 | August 2024
  
  Published by Microsoft
  
  Publication Video Download Project
- WildFeedback: Aligning LLMs With In-situ User Interactions And Feedback
  
  Taiwei Shi, Zhuoer Wang, Longqi Yang, Ying-Chun Lin, Zexue He, Mengting Wan, Pei Zhou, Sujay Kumar Jauhar, Xiaofeng Xu, Xia Song, Jennifer Neville
  
  ArXiv | August 2024, Vol abs/2408.15549
  
  Publication DOI
- Interpretable User Satisfaction Estimation for Conversational Systems with Large Language Models
  
  Ying-Chun Lin, Jennifer Neville, Jack W. Stokes, Longqi Yang, Tara Safavi, Mengting Wan, Scott Counts, Siddharth Suri, Reid Andersen, Xiaofeng Xu, Deepak Gupta, Sujay Kumar Jauhar, Xia Song, Georg Buscher, Saurabh Tiwary, Brent Hecht, Jaime Teevan
  
  ACL | August 2024
  
  Publication Project
- Model-Generated Pretraining Signals Improves Zero-Shot Generalization of Text-to-Text Transformers
  
  Linyuan Gong, Chenyan Xiong, Xiaodong Liu, Payal Bajaj, Yiqing Xie, Alvin Cheung, Jianfeng Gao, Xia Song
  
  ACL 2023 | July 2023
  
  Publication
- Magneto: A Foundation Transformer
  
  Hongyu Wang, Shuming Ma, Shaohan Huang, Li Dong, Wenhui Wang, Zhiliang Peng, Yu Wu, Payal Bajaj, Saksham Singhal, Alon Benhaim, Barun Patra, Zhun Liu, Vishrav Chaudhary, Xia Song, Furu Wei
  
  ICML 2023 | June 2023
  
  Publication
- Language Is Not All You Need: Aligning Perception with Language Models
  
  Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, Qiang Liu, Kriti Aggarwal, Zewen Chi, Johan Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, Furu Wei
  
  NeurIPS 2023 | March 2023
  
  Publication
- A Length-Extrapolatable Transformer
  
  Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, Furu Wei
  
  ACL 2023 | December 2022
  
  Publication
- Beyond English-Centric Bitexts for Better Multilingual Language Representation Learning
  
  Barun Patra, Saksham Singhal, Shaohan Huang, Zewen Chi, Li Dong, Furu Wei, Vishrav Chaudhary, Xia Song
  
  ACL 2023 | October 2022
  
  Work in progress
  
  Publication
- On the Representation Collapse of Sparse Mixture of Experts
  
  Zewen Chi, Li Dong, Shaohan Huang, Damai Dai, Shuming Ma, Barun Patra, Saksham Singhal, Payal Bajaj, Xia Song, Furu Wei
  
  2022 Neural Information Processing Systems | October 2022
  
  Publication
- METRO: Efficient Denoising Pretraining of Large Scale Autoencoding Language Models with Model Generated Signals
  
  Payal Bajaj, Chenyan Xiong, Guolin Ke, Xiaodong Liu, Saurabh Tiwary, Tie-Yan Liu, Paul Bennett, Xia Song, Jianfeng Gao
  
  April 2022
  
  arXiv:2204.06644
  
  Publication
- Pretraining Text Encoders with Adversarial Mixture of Training Signal Generators
  
  Yu Meng, Chenyan Xiong, Payal Bajaj, Saurabh Tiwary, Paul Bennett, Jiawei Han, Xia Song
  
  ICLR 2022 | April 2022
  
  Publication Github
- COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining
  
  Yu Meng, Chenyan Xiong, Payal Bajaj, Saurabh Tiwary, Paul Bennett, Jiawei Han, Xia Song
  
  NeurIPS 2021 | December 2021
  
  Publication
- Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task.
  
  Jian Yang, Shuming Ma, Haoyang Huang, Dongdong Zhang, Li Dong, Shaohan Huang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
  
  WMT | November 2021
  
  Publication Publication Project
- Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training
  
  Bo Zheng, Li Dong, Shaohan Huang, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, Furu Wei
  
  EMNLP 2021 | September 2021
  
  Publication Project
- XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
  
  Zewen Chi, Shaohan Huang, Li Dong, Shuming Ma, Saksham Singhal, Payal Bajaj, Xia Song, Furu Wei
  
  ACL 2022 | June 2021
  
  Publication Project
- DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders.
  
  Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
  
  arXiv: Computation and Language | June 2021
  
  Publication Publication Publication Project
- DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders.
  
  Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
  
  June 2021
  
  Publication Publication Project
- InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
  
  Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, Ming Zhou
  
  June 2021
  
  Publication Publication
- Consistency Regularization for Cross-Lingual Fine-Tuning
  
  Bo Zheng, Li Dong, Shaohan Huang, Wenhui Wang, Zewen Chi, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, Furu Wei
  
  ACL-IJCNLP 2021 | June 2021
  
  PDF
- Language Scaling for Universal Suggested Replies Model
  
  Qianlan Ying, Payal Bajaj, Budhaditya Deb, Yu Yang, Wei Wang, Bojia Lin, Milad Shokouhi, Xia Song, Yang Yang, Daxin Jiang (姜大昕)
  
  NAACL 2021 Industrial Track | June 2021
  
  Publication
- Knowledge-Aware Language Model Pretraining
  
  Corby Rosset, Chenyan Xiong, Minh Phan, Xia Song, Paul Bennett, Saurabh Tiwary
  
  February 2021
  
  arXiv:2007.00655
  
  Publication
- XLM-T: Scaling up Multilingual Machine Translation with Pretrained Cross-lingual Transformer Encoders.
  
  Shuming Ma, Jian Yang, Haoyang Huang, Zewen Chi, Li Dong, Dongdong Zhang, Hany Hassan Awadalla, Alexandre Muzio, Akiko I. Eriguchi, Saksham Singhal, Xia Song, Arul Menezes, Furu Wei
  
  December 2020
  
  Publication Publication Publication Project
- Pushing the Limits of Narrow Precision Inferencing at Cloud Scale with Microsoft Floating Point
  
  Bita Darvish Rouhani, Daniel Lo, Ritchie Zhao, Ming Liu, Jeremy Fowers, Kalin Ovtcharov, Anna Vinogradsky, Sarah Massengill, Lita Yang, Ray Bittner, Alessandro Forin, Haishan Zhu, Taesik Na, Prerak Patel, Shuai Che, Lok Chand Koppaka, Xia Song, Subhojit Som, Kaustav Das, Saurabh Tiwary, Steve Reinhardt, Sitaram Lanka, Eric Chung, Doug Burger
  
  NeurIPS 2020 | November 2020
  
  Publication
- InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
  
  Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, Ming Zhou
  
  July 2020
  
  Publication Publication Github
- Towards Language Agnostic Universal Representations
  
  Armen Aghajanyan, Xia Song, Saurabh Tiwary
  
  September 2018
  
  Publication Project
- Interpretable User Satisfaction Estimation for Conversational Systems with Large Language Models
  
  Ying-Chun Lin, Jennifer Neville, Jack W. Stokes, Longqi Yang, Tara Safavi, Mengting Wan, Scott Counts, Siddharth Suri, Reid Andersen, Xiaofeng Xu, Deepak Gupta, Sujay Kumar Jauhar, Xia Song, Georg Buscher, Saurabh Tiwary, Brent Hecht, Jaime Teevan
  
  ACL | August 2024
  
  Publication Project
- Model-Generated Pretraining Signals Improves Zero-Shot Generalization of Text-to-Text Transformers
  
  Linyuan Gong, Chenyan Xiong, Xiaodong Liu, Payal Bajaj, Yiqing Xie, Alvin Cheung, Jianfeng Gao, Xia Song
  
  ACL 2023 | July 2023
  
  Publication
- Magneto: A Foundation Transformer
  
  Hongyu Wang, Shuming Ma, Shaohan Huang, Li Dong, Wenhui Wang, Zhiliang Peng, Yu Wu, Payal Bajaj, Saksham Singhal, Alon Benhaim, Barun Patra, Zhun Liu, Vishrav Chaudhary, Xia Song, Furu Wei
  
  ICML 2023 | June 2023
  
  Publication
- Beyond English-Centric Bitexts for Better Multilingual Language Representation Learning
  
  Barun Patra, Saksham Singhal, Shaohan Huang, Zewen Chi, Li Dong, Furu Wei, Vishrav Chaudhary, Xia Song
  
  ACL 2023 | October 2022
  
  Work in progress
  
  Publication
- METRO: Efficient Denoising Pretraining of Large Scale Autoencoding Language Models with Model Generated Signals
  
  Payal Bajaj, Chenyan Xiong, Guolin Ke, Xiaodong Liu, Saurabh Tiwary, Tie-Yan Liu, Paul Bennett, Xia Song, Jianfeng Gao
  
  April 2022
  
  arXiv:2204.06644
  
  Publication
- Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task.
  
  Jian Yang, Shuming Ma, Haoyang Huang, Dongdong Zhang, Li Dong, Shaohan Huang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
  
  WMT | November 2021
  
  Publication Publication Project
- Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training
  
  Bo Zheng, Li Dong, Shaohan Huang, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, Furu Wei
  
  EMNLP 2021 | September 2021
  
  Publication Project
- XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
  
  Zewen Chi, Shaohan Huang, Li Dong, Shuming Ma, Saksham Singhal, Payal Bajaj, Xia Song, Furu Wei
  
  ACL 2022 | June 2021
  
  Publication Project
- DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders.
  
  Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
  
  arXiv: Computation and Language | June 2021
  
  Publication Publication Publication Project
- DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders.
  
  Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
  
  June 2021
  
  Publication Publication Project
- InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
  
  Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, Ming Zhou
  
  June 2021
  
  Publication Publication
- Knowledge-Aware Language Model Pretraining
  
  Corby Rosset, Chenyan Xiong, Minh Phan, Xia Song, Paul Bennett, Saurabh Tiwary
  
  February 2021
  
  arXiv:2007.00655
  
  Publication
- XLM-T: Scaling up Multilingual Machine Translation with Pretrained Cross-lingual Transformer Encoders.
  
  Shuming Ma, Jian Yang, Haoyang Huang, Zewen Chi, Li Dong, Dongdong Zhang, Hany Hassan Awadalla, Alexandre Muzio, Akiko I. Eriguchi, Saksham Singhal, Xia Song, Arul Menezes, Furu Wei
  
  December 2020
  
  Publication Publication Publication Project
- Magneto: A Foundation Transformer
  
  Hongyu Wang, Shuming Ma, Shaohan Huang, Li Dong, Wenhui Wang, Zhiliang Peng, Yu Wu, Payal Bajaj, Saksham Singhal, Alon Benhaim, Barun Patra, Zhun Liu, Vishrav Chaudhary, Xia Song, Furu Wei
  
  ICML 2023 | June 2023
  
  Publication
- Language Is Not All You Need: Aligning Perception with Language Models
  
  Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, Qiang Liu, Kriti Aggarwal, Zewen Chi, Johan Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, Furu Wei
  
  NeurIPS 2023 | March 2023
  
  Publication
- Pushing the Limits of Narrow Precision Inferencing at Cloud Scale with Microsoft Floating Point
  
  Bita Darvish Rouhani, Daniel Lo, Ritchie Zhao, Ming Liu, Jeremy Fowers, Kalin Ovtcharov, Anna Vinogradsky, Sarah Massengill, Lita Yang, Ray Bittner, Alessandro Forin, Haishan Zhu, Taesik Na, Prerak Patel, Shuai Che, Lok Chand Koppaka, Xia Song, Subhojit Som, Kaustav Das, Saurabh Tiwary, Steve Reinhardt, Sitaram Lanka, Eric Chung, Doug Burger
  
  NeurIPS 2020 | November 2020
  
  Publication

- POROver: Improving Safety and Reducing Overrefusal in Large Language Models with Overgeneration and Preference Optimization
  
  Batuhan K. Karaman, I. Zabir, Alon Benhaim, Vishrav Chaudhary, M. Sabuncu, Xia Song
  
  ICML 2025 | October 2024
  
  DOI Publication Publication
- Interpretable User Satisfaction Estimation for Conversational Systems with Large Language Models
  
  Ying-Chun Lin, Jennifer Neville, Jack W. Stokes, Longqi Yang, Tara Safavi, Mengting Wan, Scott Counts, Siddharth Suri, Reid Andersen, Xiaofeng Xu, Deepak Gupta, Sujay Kumar Jauhar, Xia Song, Georg Buscher, Saurabh Tiwary, Brent Hecht, Jaime Teevan
  
  ACL | August 2024
  
  Publication Project
- Model-Generated Pretraining Signals Improves Zero-Shot Generalization of Text-to-Text Transformers
  
  Linyuan Gong, Chenyan Xiong, Xiaodong Liu, Payal Bajaj, Yiqing Xie, Alvin Cheung, Jianfeng Gao, Xia Song
  
  ACL 2023 | July 2023
  
  Publication
- Magneto: A Foundation Transformer
  
  Hongyu Wang, Shuming Ma, Shaohan Huang, Li Dong, Wenhui Wang, Zhiliang Peng, Yu Wu, Payal Bajaj, Saksham Singhal, Alon Benhaim, Barun Patra, Zhun Liu, Vishrav Chaudhary, Xia Song, Furu Wei
  
  ICML 2023 | June 2023
  
  Publication
- Language Is Not All You Need: Aligning Perception with Language Models
  
  Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, Qiang Liu, Kriti Aggarwal, Zewen Chi, Johan Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, Furu Wei
  
  NeurIPS 2023 | March 2023
  
  Publication
- A Length-Extrapolatable Transformer
  
  Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, Furu Wei
  
  ACL 2023 | December 2022
  
  Publication
- Beyond English-Centric Bitexts for Better Multilingual Language Representation Learning
  
  Barun Patra, Saksham Singhal, Shaohan Huang, Zewen Chi, Li Dong, Furu Wei, Vishrav Chaudhary, Xia Song
  
  ACL 2023 | October 2022
  
  Work in progress
  
  Publication
- On the Representation Collapse of Sparse Mixture of Experts
  
  Zewen Chi, Li Dong, Shaohan Huang, Damai Dai, Shuming Ma, Barun Patra, Saksham Singhal, Payal Bajaj, Xia Song, Furu Wei
  
  2022 Neural Information Processing Systems | October 2022
  
  Publication
- Pretraining Text Encoders with Adversarial Mixture of Training Signal Generators
  
  Yu Meng, Chenyan Xiong, Payal Bajaj, Saurabh Tiwary, Paul Bennett, Jiawei Han, Xia Song
  
  ICLR 2022 | April 2022
  
  Publication Github
- COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining
  
  Yu Meng, Chenyan Xiong, Payal Bajaj, Saurabh Tiwary, Paul Bennett, Jiawei Han, Xia Song
  
  NeurIPS 2021 | December 2021
  
  Publication
- Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task.
  
  Jian Yang, Shuming Ma, Haoyang Huang, Dongdong Zhang, Li Dong, Shaohan Huang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
  
  WMT | November 2021
  
  Publication Publication Project
- Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training
  
  Bo Zheng, Li Dong, Shaohan Huang, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, Furu Wei
  
  EMNLP 2021 | September 2021
  
  Publication Project
- XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
  
  Zewen Chi, Shaohan Huang, Li Dong, Shuming Ma, Saksham Singhal, Payal Bajaj, Xia Song, Furu Wei
  
  ACL 2022 | June 2021
  
  Publication Project
- InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
  
  Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, Ming Zhou
  
  June 2021
  
  Publication Publication
- Consistency Regularization for Cross-Lingual Fine-Tuning
  
  Bo Zheng, Li Dong, Shaohan Huang, Wenhui Wang, Zewen Chi, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, Furu Wei
  
  ACL-IJCNLP 2021 | June 2021
  
  PDF
- Language Scaling for Universal Suggested Replies Model
  
  Qianlan Ying, Payal Bajaj, Budhaditya Deb, Yu Yang, Wei Wang, Bojia Lin, Milad Shokouhi, Xia Song, Yang Yang, Daxin Jiang (姜大昕)
  
  NAACL 2021 Industrial Track | June 2021
  
  Publication
- Pushing the Limits of Narrow Precision Inferencing at Cloud Scale with Microsoft Floating Point
  
  Bita Darvish Rouhani, Daniel Lo, Ritchie Zhao, Ming Liu, Jeremy Fowers, Kalin Ovtcharov, Anna Vinogradsky, Sarah Massengill, Lita Yang, Ray Bittner, Alessandro Forin, Haishan Zhu, Taesik Na, Prerak Patel, Shuai Che, Lok Chand Koppaka, Xia Song, Subhojit Som, Kaustav Das, Saurabh Tiwary, Steve Reinhardt, Sitaram Lanka, Eric Chung, Doug Burger
  
  NeurIPS 2020 | November 2020
  
  Publication
- WildFeedback: Aligning LLMs With In-situ User Interactions And Feedback
  
  Taiwei Shi, Zhuoer Wang, Longqi Yang, Ying-Chun Lin, Zexue He, Mengting Wan, Pei Zhou, Sujay Kumar Jauhar, Xiaofeng Xu, Xia Song, Jennifer Neville
  
  ArXiv | August 2024, Vol abs/2408.15549
  
  Publication DOI
- DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders.
  
  Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
  
  arXiv: Computation and Language | June 2021
  
  Publication Publication Publication Project
- Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
  
  Marah I Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Hassan Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Caio César Teodoro Mendes, Weizhu Chen, Vishrav Chaudhary, Parul Chopra, Allie Del Giorno, Gustavo de Rosa, Matthew Dixon, Ronen Eldan, Dan Iter, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Jamie Huynh, Mojan Javaheripi, Xin Jin, Piero Kauffmann, Nikos Karampatziakis, Dongwoo Kim, Mahmoud Khademi, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Chen Liang, Weishung Liu, Xihui (Eric) Lin, Zeqi Lin, Piyush Madan, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez-Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac, Corby Rosset, Sambudha Roy, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang, Hiteshi Sharma, Xia Song, Olatunji Ruwase, Xin Wang, Rachel Ward, Guanhua Wang, Philipp Witte, Michael Wyatt, Can Xu, Jiahang Xu, Weijian Xu, Sonali Yadav, Fan Yang, Ziyi Yang, Donghan Yu, Chengruidong Zhang, Cyril Zhang, Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yunan Zhang, Xiren Zhou
  
  MSR-TR-2024-12 | August 2024
  
  Published by Microsoft
  
  Publication Video Download Project
- METRO: Efficient Denoising Pretraining of Large Scale Autoencoding Language Models with Model Generated Signals
  
  Payal Bajaj, Chenyan Xiong, Guolin Ke, Xiaodong Liu, Saurabh Tiwary, Tie-Yan Liu, Paul Bennett, Xia Song, Jianfeng Gao
  
  April 2022
  
  arXiv:2204.06644
  
  Publication
- DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders.
  
  Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
  
  June 2021
  
  Publication Publication Project
- Knowledge-Aware Language Model Pretraining
  
  Corby Rosset, Chenyan Xiong, Minh Phan, Xia Song, Paul Bennett, Saurabh Tiwary
  
  February 2021
  
  arXiv:2007.00655
  
  Publication
- XLM-T: Scaling up Multilingual Machine Translation with Pretrained Cross-lingual Transformer Encoders.
  
  Shuming Ma, Jian Yang, Haoyang Huang, Zewen Chi, Li Dong, Dongdong Zhang, Hany Hassan Awadalla, Alexandre Muzio, Akiko I. Eriguchi, Saksham Singhal, Xia Song, Arul Menezes, Furu Wei
  
  December 2020
  
  Publication Publication Publication Project
- InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
  
  Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, Ming Zhou
  
  July 2020
  
  Publication Publication Github
- Towards Language Agnostic Universal Representations
  
  Armen Aghajanyan, Xia Song, Saurabh Tiwary
  
  September 2018
  
  Publication Project