Shaohan Huang

Senior Researcher

Papiers de Collaboration

Afficher par:

- REDSTONE: Curating General, Code, Math, and QA Data for Large Language Models
  
  Yaoyao Chang, Lei Cui, Li Dong, Shaohan Huang, Yangyu Huang, Yupan Huang, Scarlett Li, Tengchao Lv, Shuming Ma, Qinzheng Sun, Wenhui Wang, Furu Wei, Ying Xin, Mao Yang, Qiufeng Yin, Xingxing Zhang
  
  December 2024
  
  Publication
- Boosting Text-to-Video Generative Model with MLLMs Feedback
  
  Xun Wu, Shaohan Huang, Furu Wei
  
  October 2024
  
  Publication
- Multimodal Large Language Models Make Text-to-Image Generative Models Align Better
  
  Xun Wu, Shaohan Huang, Furu Wei
  
  October 2024
  
  Publication
- Instruction Pre-Training: Language Models are Supervised Multitask Learners
  
  Daixuan Cheng, Yuxian Gu, Shaohan Huang, Junyu Bi, Minlie Huang, Furu Wei
  
  June 2024
  
  DOI Publication Publication
- MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning
  
  Ting Jiang, Shaohan Huang, Shengyue Luo, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, Fuzhen Zhuang
  
  May 2024
  
  DOI Publication Publication
- You Only Cache Once: Decoder-Decoder Architectures for Language Models
  
  Yutao Sun, Li Dong, Yi Zhu, Shaohan Huang, Wenhui Wang, Shuming Ma, Quanlu Zhang, Jianyong Wang, Furu Wei
  
  NeurIPS 2024 | May 2024
  
  DOI Publication Publication
- Kosmos-G: Generating Images in Context with Multimodal Large Language Models
  
  Xichen Pan, Li Dong, Shaohan Huang, Zhiliang Peng, Wenhu Chen, Furu Wei
  
  ICLR 2024 | May 2024
  
  DOI Publication PDF
- Multi-Head Mixture-of-Experts
  
  Xun Wu, Shaohan Huang, Wenhui Wang, Furu Wei
  
  NeurIPS 2024 | April 2024
  
  DOI Publication Publication
- Mixture of LoRA Experts
  
  Shaohan Huang, Furu Wei
  
  ICLR 2024 | April 2024
  
  Publication
- ResLoRA: Identity Residual Mapping in Low-Rank Adaption
  
  Shuhua Shi, Shaohan Huang, Minghui Song, Zhoujun Li, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang
  
  February 2024
  
  DOI Publication Publication
- The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
  
  Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang, Shaohan Huang, Lifeng Dong, Ruiping Wang, Jilong Xue, Furu Wei
  
  February 2024
  
  Work in progress
  
  DOI Publication Publication
- Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models
  
  Haoran Li, Qingxiu Dong, Zhengyang Tang, Chaojun Wang, Xingxing Zhang, Haoyang Huang, Shaohan Huang, Xiaolong Huang, Zeqiang Huang, Dongdong Zhang, Yuxian Gu, Xin Cheng, Xun Wang, Si-Qing Chen, Li Dong, Wei Lu, Zhifang Sui, Benyou Wang, Wai Lam, Furu Wei
  
  February 2024
  
  Work in progress
  
  DOI Publication Publication
- Text Diffusion with Reinforced Conditioning
  
  Yuxuan Liu, Tianchi Yang, Shaohan Huang, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang
  
  AAAI 2024 | February 2024
  
  Publication
- BitNet: Scaling 1-bit Transformers for Large Language Models
  
  Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang, Yi Wu, Furu Wei
  
  October 2023
  
  DOI Publication
- Adapting Large Language Models via Reading Comprehension
  
  Daixuan Cheng, Shaohan Huang, Furu Wei
  
  ICLR 2024 | September 2023
  
  DOI
- Kosmos-2.5: A Multimodal Literate Model
  
  Tengchao Lv, Yupan Huang, Jingye Chen, Lei Cui, Shuming Ma, Yaoyao Chang, Shaohan Huang, Wenhui Wang, Li Dong, Weiyao Luo, Shaoxiang Wu, Guoxin Wang, Cha Zhang, Furu Wei
  
  September 2023
  
  Preprint Projet
- Retentive Network: A Successor to Transformer for Large Language Models
  
  Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, Furu Wei
  
  August 2023
  
  Publication
- LongNet: Scaling Transformers to 1,000,000,000 Tokens
  
  Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, Furu Wei
  
  July 2023
  
  DOI PDF Vidéo
- Language Is Not All You Need: Aligning Perception with Language Models
  
  Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, Qiang Liu, Kriti Aggarwal, Zewen Chi, Johan Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, Furu Wei
  
  NeurIPS 2023 | March 2023
  
  Publication
- Beyond English-Centric Bitexts for Better Multilingual Language Representation Learning
  
  Barun Patra, Saksham Singhal, Shaohan Huang, Zewen Chi, Li Dong, Furu Wei, Vishrav Chaudhary, Xia Song
  
  ACL 2023 | October 2022
  
  Work in progress
  
  Publication
- THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption
  
  Tianyu Chen, Hangbo Bao, Shaohan Huang, Li Dong, Binxing Jiao, Daxin Jiang (姜大昕), Haoyi Zhou, Jianxin Li, Furu Wei
  
  May 2022
- Learning Music Sequence Representation from Text Supervision
  
  Tianyu Chen, Shuai Zhang, Haoyi Zhou, Jianxin Li, Yuan Xie, Shaohan Huang
  
  April 2022
- Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task.
  
  Jian Yang, Shuming Ma, Haoyang Huang, Dongdong Zhang, Li Dong, Shaohan Huang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
  
  WMT | November 2021
  
  Publication Publication Projet
- MT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs
  
  Zewen Chi, Li Dong, Shuming Ma, Shaohan Huang, Xian-Ling Mao, Heyan Huang, Furu Wei
  
  EMNLP 2021 | October 2021
  
  Publication
- Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training
  
  Bo Zheng, Li Dong, Shaohan Huang, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, Furu Wei
  
  EMNLP 2021 | September 2021
  
  Publication Projet
- Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains
  
  Yunzhi Yao, Shaohan Huang, Wenhui Wang, Li Dong, Furu Wei
  
  ACL-IJCNLP 2021 | July 2021
  
  Publication Github
- XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
  
  Zewen Chi, Shaohan Huang, Li Dong, Shuming Ma, Saksham Singhal, Payal Bajaj, Xia Song, Furu Wei
  
  ACL 2022 | June 2021
  
  Publication Projet
- DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders.
  
  Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
  
  arXiv: Computation and Language | June 2021
  
  Publication Publication Publication Projet
- DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders.
  
  Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
  
  June 2021
  
  Publication Publication Projet
- Consistency Regularization for Cross-Lingual Fine-Tuning
  
  Bo Zheng, Li Dong, Shaohan Huang, Wenhui Wang, Zewen Chi, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, Furu Wei
  
  ACL-IJCNLP 2021 | June 2021
  
  PDF
- Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
  
  Zewen Chi, Li Dong, Bo Zheng, Shaohan Huang, Xian-Ling Mao, Heyan Huang, Furu Wei
  
  ACL-IJCNLP 2021 | June 2021
  
  Publication
- MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers
  
  Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, Furu Wei
  
  ACL-IJCNLP 2021 | December 2020
  
  PDF Publication
- LayoutLM: Pre-training of Text and Layout for Document Image Understanding
  
  Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, Ming Zhou
  
  KDD 2020 | August 2020
  
  PDF Publication Github Projet

- REDSTONE: Curating General, Code, Math, and QA Data for Large Language Models
  
  Yaoyao Chang, Lei Cui, Li Dong, Shaohan Huang, Yangyu Huang, Yupan Huang, Scarlett Li, Tengchao Lv, Shuming Ma, Qinzheng Sun, Wenhui Wang, Furu Wei, Ying Xin, Mao Yang, Qiufeng Yin, Xingxing Zhang
  
  December 2024
  
  Publication
- Boosting Text-to-Video Generative Model with MLLMs Feedback
  
  Xun Wu, Shaohan Huang, Furu Wei
  
  October 2024
  
  Publication
- Instruction Pre-Training: Language Models are Supervised Multitask Learners
  
  Daixuan Cheng, Yuxian Gu, Shaohan Huang, Junyu Bi, Minlie Huang, Furu Wei
  
  June 2024
  
  DOI Publication Publication
- MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning
  
  Ting Jiang, Shaohan Huang, Shengyue Luo, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, Fuzhen Zhuang
  
  May 2024
  
  DOI Publication Publication
- You Only Cache Once: Decoder-Decoder Architectures for Language Models
  
  Yutao Sun, Li Dong, Yi Zhu, Shaohan Huang, Wenhui Wang, Shuming Ma, Quanlu Zhang, Jianyong Wang, Furu Wei
  
  NeurIPS 2024 | May 2024
  
  DOI Publication Publication
- Kosmos-G: Generating Images in Context with Multimodal Large Language Models
  
  Xichen Pan, Li Dong, Shaohan Huang, Zhiliang Peng, Wenhu Chen, Furu Wei
  
  ICLR 2024 | May 2024
  
  DOI Publication PDF
- Multi-Head Mixture-of-Experts
  
  Xun Wu, Shaohan Huang, Wenhui Wang, Furu Wei
  
  NeurIPS 2024 | April 2024
  
  DOI Publication Publication
- ResLoRA: Identity Residual Mapping in Low-Rank Adaption
  
  Shuhua Shi, Shaohan Huang, Minghui Song, Zhoujun Li, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang
  
  February 2024
  
  DOI Publication Publication
- The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
  
  Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang, Shaohan Huang, Lifeng Dong, Ruiping Wang, Jilong Xue, Furu Wei
  
  February 2024
  
  Work in progress
  
  DOI Publication Publication
- Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models
  
  Haoran Li, Qingxiu Dong, Zhengyang Tang, Chaojun Wang, Xingxing Zhang, Haoyang Huang, Shaohan Huang, Xiaolong Huang, Zeqiang Huang, Dongdong Zhang, Yuxian Gu, Xin Cheng, Xun Wang, Si-Qing Chen, Li Dong, Wei Lu, Zhifang Sui, Benyou Wang, Wai Lam, Furu Wei
  
  February 2024
  
  Work in progress
  
  DOI Publication Publication
- Text Diffusion with Reinforced Conditioning
  
  Yuxuan Liu, Tianchi Yang, Shaohan Huang, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang
  
  AAAI 2024 | February 2024
  
  Publication
- BitNet: Scaling 1-bit Transformers for Large Language Models
  
  Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang, Yi Wu, Furu Wei
  
  October 2023
  
  DOI Publication
- Adapting Large Language Models via Reading Comprehension
  
  Daixuan Cheng, Shaohan Huang, Furu Wei
  
  ICLR 2024 | September 2023
  
  DOI
- Kosmos-2.5: A Multimodal Literate Model
  
  Tengchao Lv, Yupan Huang, Jingye Chen, Lei Cui, Shuming Ma, Yaoyao Chang, Shaohan Huang, Wenhui Wang, Li Dong, Weiyao Luo, Shaoxiang Wu, Guoxin Wang, Cha Zhang, Furu Wei
  
  September 2023
  
  Preprint Projet
- Retentive Network: A Successor to Transformer for Large Language Models
  
  Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, Furu Wei
  
  August 2023
  
  Publication
- LongNet: Scaling Transformers to 1,000,000,000 Tokens
  
  Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, Furu Wei
  
  July 2023
  
  DOI PDF Vidéo
- Language Is Not All You Need: Aligning Perception with Language Models
  
  Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, Qiang Liu, Kriti Aggarwal, Zewen Chi, Johan Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, Furu Wei
  
  NeurIPS 2023 | March 2023
  
  Publication
- Beyond English-Centric Bitexts for Better Multilingual Language Representation Learning
  
  Barun Patra, Saksham Singhal, Shaohan Huang, Zewen Chi, Li Dong, Furu Wei, Vishrav Chaudhary, Xia Song
  
  ACL 2023 | October 2022
  
  Work in progress
  
  Publication
- Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task.
  
  Jian Yang, Shuming Ma, Haoyang Huang, Dongdong Zhang, Li Dong, Shaohan Huang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
  
  WMT | November 2021
  
  Publication Publication Projet
- Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training
  
  Bo Zheng, Li Dong, Shaohan Huang, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, Furu Wei
  
  EMNLP 2021 | September 2021
  
  Publication Projet
- Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains
  
  Yunzhi Yao, Shaohan Huang, Wenhui Wang, Li Dong, Furu Wei
  
  ACL-IJCNLP 2021 | July 2021
  
  Publication Github
- XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
  
  Zewen Chi, Shaohan Huang, Li Dong, Shuming Ma, Saksham Singhal, Payal Bajaj, Xia Song, Furu Wei
  
  ACL 2022 | June 2021
  
  Publication Projet
- DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders.
  
  Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
  
  arXiv: Computation and Language | June 2021
  
  Publication Publication Publication Projet
- DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders.
  
  Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
  
  June 2021
  
  Publication Publication Projet
- Consistency Regularization for Cross-Lingual Fine-Tuning
  
  Bo Zheng, Li Dong, Shaohan Huang, Wenhui Wang, Zewen Chi, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, Furu Wei
  
  ACL-IJCNLP 2021 | June 2021
  
  PDF
- Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
  
  Zewen Chi, Li Dong, Bo Zheng, Shaohan Huang, Xian-Ling Mao, Heyan Huang, Furu Wei
  
  ACL-IJCNLP 2021 | June 2021
  
  Publication
- MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers
  
  Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, Furu Wei
  
  ACL-IJCNLP 2021 | December 2020
  
  PDF Publication
- LayoutLM: Pre-training of Text and Layout for Document Image Understanding
  
  Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, Ming Zhou
  
  KDD 2020 | August 2020
  
  PDF Publication Github Projet
- Multimodal Large Language Models Make Text-to-Image Generative Models Align Better
  
  Xun Wu, Shaohan Huang, Furu Wei
  
  October 2024
  
  Publication
- Kosmos-2.5: A Multimodal Literate Model
  
  Tengchao Lv, Yupan Huang, Jingye Chen, Lei Cui, Shuming Ma, Yaoyao Chang, Shaohan Huang, Wenhui Wang, Li Dong, Weiyao Luo, Shaoxiang Wu, Guoxin Wang, Cha Zhang, Furu Wei
  
  September 2023
  
  Preprint Projet
- Language Is Not All You Need: Aligning Perception with Language Models
  
  Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, Qiang Liu, Kriti Aggarwal, Zewen Chi, Johan Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, Furu Wei
  
  NeurIPS 2023 | March 2023
  
  Publication
- Mixture of LoRA Experts
  
  Shaohan Huang, Furu Wei
  
  ICLR 2024 | April 2024
  
  Publication
- Kosmos-2.5: A Multimodal Literate Model
  
  Tengchao Lv, Yupan Huang, Jingye Chen, Lei Cui, Shuming Ma, Yaoyao Chang, Shaohan Huang, Wenhui Wang, Li Dong, Weiyao Luo, Shaoxiang Wu, Guoxin Wang, Cha Zhang, Furu Wei
  
  September 2023
  
  Preprint Projet
- Beyond English-Centric Bitexts for Better Multilingual Language Representation Learning
  
  Barun Patra, Saksham Singhal, Shaohan Huang, Zewen Chi, Li Dong, Furu Wei, Vishrav Chaudhary, Xia Song
  
  ACL 2023 | October 2022
  
  Work in progress
  
  Publication
- THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption
  
  Tianyu Chen, Hangbo Bao, Shaohan Huang, Li Dong, Binxing Jiao, Daxin Jiang (姜大昕), Haoyi Zhou, Jianxin Li, Furu Wei
  
  May 2022
- Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task.
  
  Jian Yang, Shuming Ma, Haoyang Huang, Dongdong Zhang, Li Dong, Shaohan Huang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
  
  WMT | November 2021
  
  Publication Publication Projet
- MT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs
  
  Zewen Chi, Li Dong, Shuming Ma, Shaohan Huang, Xian-Ling Mao, Heyan Huang, Furu Wei
  
  EMNLP 2021 | October 2021
  
  Publication
- Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training
  
  Bo Zheng, Li Dong, Shaohan Huang, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, Furu Wei
  
  EMNLP 2021 | September 2021
  
  Publication Projet
- XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
  
  Zewen Chi, Shaohan Huang, Li Dong, Shuming Ma, Saksham Singhal, Payal Bajaj, Xia Song, Furu Wei
  
  ACL 2022 | June 2021
  
  Publication Projet
- DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders.
  
  Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
  
  arXiv: Computation and Language | June 2021
  
  Publication Publication Publication Projet
- DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders.
  
  Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
  
  June 2021
  
  Publication Publication Projet
- Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
  
  Zewen Chi, Li Dong, Bo Zheng, Shaohan Huang, Xian-Ling Mao, Heyan Huang, Furu Wei
  
  ACL-IJCNLP 2021 | June 2021
  
  Publication
- LayoutLM: Pre-training of Text and Layout for Document Image Understanding
  
  Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, Ming Zhou
  
  KDD 2020 | August 2020
  
  PDF Publication Github Projet
- Learning Music Sequence Representation from Text Supervision
  
  Tianyu Chen, Shuai Zhang, Haoyi Zhou, Jianxin Li, Yuan Xie, Shaohan Huang
  
  April 2022

- REDSTONE: Curating General, Code, Math, and QA Data for Large Language Models
  
  Yaoyao Chang, Lei Cui, Li Dong, Shaohan Huang, Yangyu Huang, Yupan Huang, Scarlett Li, Tengchao Lv, Shuming Ma, Qinzheng Sun, Wenhui Wang, Furu Wei, Ying Xin, Mao Yang, Qiufeng Yin, Xingxing Zhang
  
  December 2024
  
  Publication
- Multimodal Large Language Models Make Text-to-Image Generative Models Align Better
  
  Xun Wu, Shaohan Huang, Furu Wei
  
  October 2024
  
  Publication
- Boosting Text-to-Video Generative Model with MLLMs Feedback
  
  Xun Wu, Shaohan Huang, Furu Wei
  
  October 2024
  
  Publication
- Kosmos-2.5: A Multimodal Literate Model
  
  Tengchao Lv, Yupan Huang, Jingye Chen, Lei Cui, Shuming Ma, Yaoyao Chang, Shaohan Huang, Wenhui Wang, Li Dong, Weiyao Luo, Shaoxiang Wu, Guoxin Wang, Cha Zhang, Furu Wei
  
  September 2023
  
  Preprint Projet
- THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption
  
  Tianyu Chen, Hangbo Bao, Shaohan Huang, Li Dong, Binxing Jiao, Daxin Jiang (姜大昕), Haoyi Zhou, Jianxin Li, Furu Wei
  
  May 2022
- Learning Music Sequence Representation from Text Supervision
  
  Tianyu Chen, Shuai Zhang, Haoyi Zhou, Jianxin Li, Yuan Xie, Shaohan Huang
  
  April 2022
- DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders.
  
  Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
  
  June 2021
  
  Publication Publication Projet
- Instruction Pre-Training: Language Models are Supervised Multitask Learners
  
  Daixuan Cheng, Yuxian Gu, Shaohan Huang, Junyu Bi, Minlie Huang, Furu Wei
  
  June 2024
  
  DOI Publication Publication
- MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning
  
  Ting Jiang, Shaohan Huang, Shengyue Luo, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, Fuzhen Zhuang
  
  May 2024
  
  DOI Publication Publication
- ResLoRA: Identity Residual Mapping in Low-Rank Adaption
  
  Shuhua Shi, Shaohan Huang, Minghui Song, Zhoujun Li, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang
  
  February 2024
  
  DOI Publication Publication
- The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
  
  Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang, Shaohan Huang, Lifeng Dong, Ruiping Wang, Jilong Xue, Furu Wei
  
  February 2024
  
  Work in progress
  
  DOI Publication Publication
- Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models
  
  Haoran Li, Qingxiu Dong, Zhengyang Tang, Chaojun Wang, Xingxing Zhang, Haoyang Huang, Shaohan Huang, Xiaolong Huang, Zeqiang Huang, Dongdong Zhang, Yuxian Gu, Xin Cheng, Xun Wang, Si-Qing Chen, Li Dong, Wei Lu, Zhifang Sui, Benyou Wang, Wai Lam, Furu Wei
  
  February 2024
  
  Work in progress
  
  DOI Publication Publication
- BitNet: Scaling 1-bit Transformers for Large Language Models
  
  Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang, Yi Wu, Furu Wei
  
  October 2023
  
  DOI Publication
- Retentive Network: A Successor to Transformer for Large Language Models
  
  Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, Furu Wei
  
  August 2023
  
  Publication
- LongNet: Scaling Transformers to 1,000,000,000 Tokens
  
  Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, Furu Wei
  
  July 2023
  
  DOI PDF Vidéo
- You Only Cache Once: Decoder-Decoder Architectures for Language Models
  
  Yutao Sun, Li Dong, Yi Zhu, Shaohan Huang, Wenhui Wang, Shuming Ma, Quanlu Zhang, Jianyong Wang, Furu Wei
  
  NeurIPS 2024 | May 2024
  
  DOI Publication Publication
- Kosmos-G: Generating Images in Context with Multimodal Large Language Models
  
  Xichen Pan, Li Dong, Shaohan Huang, Zhiliang Peng, Wenhu Chen, Furu Wei
  
  ICLR 2024 | May 2024
  
  DOI Publication PDF
- Multi-Head Mixture-of-Experts
  
  Xun Wu, Shaohan Huang, Wenhui Wang, Furu Wei
  
  NeurIPS 2024 | April 2024
  
  DOI Publication Publication
- Mixture of LoRA Experts
  
  Shaohan Huang, Furu Wei
  
  ICLR 2024 | April 2024
  
  Publication
- Text Diffusion with Reinforced Conditioning
  
  Yuxuan Liu, Tianchi Yang, Shaohan Huang, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang
  
  AAAI 2024 | February 2024
  
  Publication
- Adapting Large Language Models via Reading Comprehension
  
  Daixuan Cheng, Shaohan Huang, Furu Wei
  
  ICLR 2024 | September 2023
  
  DOI
- Language Is Not All You Need: Aligning Perception with Language Models
  
  Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, Qiang Liu, Kriti Aggarwal, Zewen Chi, Johan Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, Furu Wei
  
  NeurIPS 2023 | March 2023
  
  Publication
- Beyond English-Centric Bitexts for Better Multilingual Language Representation Learning
  
  Barun Patra, Saksham Singhal, Shaohan Huang, Zewen Chi, Li Dong, Furu Wei, Vishrav Chaudhary, Xia Song
  
  ACL 2023 | October 2022
  
  Work in progress
  
  Publication
- Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task.
  
  Jian Yang, Shuming Ma, Haoyang Huang, Dongdong Zhang, Li Dong, Shaohan Huang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
  
  WMT | November 2021
  
  Publication Publication Projet
- MT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs
  
  Zewen Chi, Li Dong, Shuming Ma, Shaohan Huang, Xian-Ling Mao, Heyan Huang, Furu Wei
  
  EMNLP 2021 | October 2021
  
  Publication
- Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training
  
  Bo Zheng, Li Dong, Shaohan Huang, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, Furu Wei
  
  EMNLP 2021 | September 2021
  
  Publication Projet
- Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains
  
  Yunzhi Yao, Shaohan Huang, Wenhui Wang, Li Dong, Furu Wei
  
  ACL-IJCNLP 2021 | July 2021
  
  Publication Github
- XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
  
  Zewen Chi, Shaohan Huang, Li Dong, Shuming Ma, Saksham Singhal, Payal Bajaj, Xia Song, Furu Wei
  
  ACL 2022 | June 2021
  
  Publication Projet
- Consistency Regularization for Cross-Lingual Fine-Tuning
  
  Bo Zheng, Li Dong, Shaohan Huang, Wenhui Wang, Zewen Chi, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, Furu Wei
  
  ACL-IJCNLP 2021 | June 2021
  
  PDF
- Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
  
  Zewen Chi, Li Dong, Bo Zheng, Shaohan Huang, Xian-Ling Mao, Heyan Huang, Furu Wei
  
  ACL-IJCNLP 2021 | June 2021
  
  Publication
- MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers
  
  Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, Furu Wei
  
  ACL-IJCNLP 2021 | December 2020
  
  PDF Publication
- LayoutLM: Pre-training of Text and Layout for Document Image Understanding
  
  Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, Ming Zhou
  
  KDD 2020 | August 2020
  
  PDF Publication Github Projet
- DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders.
  
  Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
  
  arXiv: Computation and Language | June 2021
  
  Publication Publication Publication Projet