Self-Evolved Reward Learning for LLMs

Chenghua Huang1, Zhizhen Fan2, Lu Wang3, Fangkai Yang3, Pu Zhao3, Zeqi Lin3
Qingwei Lin3, Dongmei Zhang3, Saravan Rajmohan3, Qi Zhang3
1School of Computer Science, Fudan University
2School of Computer Science, Peking University
3Microsoft
Email: huangch22@m.fudan.edu.cn, 2201210191@stu.pku.edu.cn
wlu@microsoft.com, fangkaiyang@microsoft.com, puzhao@microsoft.com
arXiv Code Paper HuggingFace Icon HuggingFace-Coming Soon
MY ALT TEXT

The Self-Evolved Reward Learning (SER) pipeline. Our SER method consists of following steps: (1) Self-labeling: the reward model (RM) assigns labels to unlabeled data. (2) Identifying learning status and selecting data: high-confidence data is selected by assessing the learning status. (3) Retrain the RM: the RM trains itself using the self-labeled and selected data. (4) Train the Large Language Model (LLM): the LLM is trained under the guidance of the self-evolved RM. Note that steps (1)-(3) iterate multiple rounds to a converged RM.

Abstract

Reinforcement Learning from Human Feedback (RLHF) is a crucial technique for aligning language models with human preferences, playing a pivotal role in the success of conversational models like GPT-4, ChatGPT, and Llama 2. A core challenge in employing RLHF lies in training a reliable reward model (RM), which relies on high-quality labels typically provided by human experts or advanced AI system. These methods can be costly and may introduce biases that affect the language model's responses. As language models improve, human input may become less effective in further enhancing their performance. In this paper, we propose Self-Evolved Reward Learning (SER), a novel approach where the RM generates additional training data to iteratively improve itself. We conducted extensive experiments on multiple datasets such as HH-RLHF and UltraFeedback, using models like Mistral and Llama 3, and compare SER against various baselines. Our results demonstrate that even with limited human-annotated data, learning from self-feedback can robustly enhance RM performance, thereby boosting the capabilities of large language models (LLMs).

Experiment

MY ALT TEXT

Reward modeling improves in performance with iterative evolution. We demonstrate the performance variation of the model during the iterative process on the HH-RLHF, Ultrafeedback, and Summarize datasets. Baseline refers to the RM that uses the full dataset of human-annotated data.

BibTeX

@article{huang2024self,
  title={Self-Evolved Reward Learning for LLMs},
  author={Huang, Chenghua and Fan, Zhizhen and Wang, Lu and Yang, Fangkai and Zhao, Pu and Lin, Zeqi and Lin, Qingwei and Zhang, Dongmei and Rajmohan, Saravan and Zhang, Qi},
  journal={arXiv preprint arXiv:2411.00418},
  year={2024}
}