Performance evaluation and improvement of deep Q network for lunar landing task

深度Q网络在月球着陆任务中的性能评估与改进

¹ School of Automation, Xi'an University of Posts & Telecommunications, Xi'an 710121, China
² School of Astronautics, Northwestern Polytechnical University, Xi'an 710072, China

Received: 30 May 2023

Abstract

Reinforcement learning is now being applied more and more in a variety of scenarios, the majority of which are based on the deep Q network (DQN) technology. However, the algorithm is heavily influenced by multiple factors. In this paper, we take the lunar lander as a case to study how various hyper-parameters affect the performance of the DQN algorithm, based on which we tune to get a model with better performance. At present, it is known that the DQN model has an average reward of 280+ on 100 test episodes, and the reward value of the model in this article can reach 290+. Meanwhile, its robustness is tested and verified by introducing additional uncertainty tests into the original problem. In addition, to speed up the training process, imitation learning is incorporated in our model, using heuristic function model guidance method to obtain demonstration data, which accelerates training speed and improves performance. Simulation results have proven the effectiveness of this method.

摘要

基于深度Q网络(DQN)技术的强化学习方法得到越来越广泛的应用, 但该类算法的性能深受多因素影响。文中以月球登陆器为例, 探讨不同超参数对DQN性能的影响, 在此基础上训练得到性能较优的模型。目前已知DQN模型在100个测试回合下平均奖励为280+, 文中模型奖励值可达到290+, 并且通过在原始问题中引入额外的不确定性测试验证了文中模型的鲁棒性。另外, 引入模仿学习的思想, 基于启发式函数的模型指导方法获取演示数据, 加快训练速度并提升性能, 仿真结果证明了该方法的有效性。