大语言模型中的上下文面板调节与奖励模型
Journal: Advances in Computer and Autonomous Intelligence Research DOI: 10.12238/acair.v1i3.6550
Abstract
直接偏好优化(DPO)旨在符合人类偏好,同时减少强化学习的复杂性。传统方法如人类反馈强化学习(RLHF)首先匹配奖励模型与提示和偏好,然后使用强化学习(RL)来找到最大化奖励的策略。相比之下,DPO通过直接优化策略来满足偏好,无需显式奖励函数或强化学习,简化了过程。DPO是微调语言模型以保持与人类反馈一致的更直接、更有效的方法。此外,OpenAI提到他们通过模仿人类评分来训练模型,以帮助改善RLHF。下一步是将模型拟合到含有丰富“条件”的数据集上,例如训练模型生成包含记忆、条件、目标、计划、未来任务的面板,并使用这个面板进行训练。这些条件将“创意写作任务”转变为“分配材料”的任务,减少了创意写作中的熵。条件强化学习微调(C-RLFT)使得大语言模型能够理解和生成类人文本、适应新信息和个性化响应,同时保持相关性和连贯性。未来的改进工作包括使用RLHF或RLAIF改善条件面板、数据集和模型之间的迭代、使模型与现实世界需求保持一致,以及基于0阶优化构建新的基础模型。这些方向旨在使大语言模型更高效、符合人类偏好,并能在各种环境中运行,包括边缘计算设备。
Keywords
直接偏好优化；人类反馈强化学习；条件面板；创意写作熵降低；C-RLFT训练；边缘计算
Full Text
PDF - Viewed/Downloaded: 0 TimesReferences
[1] EduardoG Altmann,Giampaolo Cristadoro,and Mirko Degl iEsposti. On the origin of long range correlations in texts. Proceedings of the National Academy of Sciences,109(29):11582 -11587,2012.25.
[2] Gehman,S.Gururangan,M.Sap,Y.Choi,and N.A.Smith. RealT oxicityPrompts:Evaluatingneural toxic degeneration in langu age models. In Findings of the Association for Computational Linguistics:EMNLP 2020,pages3356-3369,Online,Nov.2020.Assoc iation for Computational Linguistics.doi:10.18653/v1/ 2020.findings-emnlp.301.URL https://aclanthology.org/2.
[3] Jared Kaplan, Sam McCandlish, Tom Henighan,Tom B Bro wn,Benjamin Chess, Rewon Child,Scott Gray, Alec Radford,Jeff rey Wu,and Dario Amodei.“Scaling laws for neural language mo dels”.In:arXiv preprint arXiv:2001.08361 (2020).020.findings -emnlp.301.
[2] Gehman,S.Gururangan,M.Sap,Y.Choi,and N.A.Smith. RealT oxicityPrompts:Evaluatingneural toxic degeneration in langu age models. In Findings of the Association for Computational Linguistics:EMNLP 2020,pages3356-3369,Online,Nov.2020.Assoc iation for Computational Linguistics.doi:10.18653/v1/ 2020.findings-emnlp.301.URL https://aclanthology.org/2.
[3] Jared Kaplan, Sam McCandlish, Tom Henighan,Tom B Bro wn,Benjamin Chess, Rewon Child,Scott Gray, Alec Radford,Jeff rey Wu,and Dario Amodei.“Scaling laws for neural language mo dels”.In:arXiv preprint arXiv:2001.08361 (2020).020.findings -emnlp.301.
Copyright © 2023 文木源
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License