The deep deterministic policy gradient(DDPG)algo-rithm is an off-policy method that combines two mainstream reinforcement learning methods based on value iteration and policy iteration.Using the DDPG algorithm,agents ...The deep deterministic policy gradient(DDPG)algo-rithm is an off-policy method that combines two mainstream reinforcement learning methods based on value iteration and policy iteration.Using the DDPG algorithm,agents can explore and summarize the environment to achieve autonomous deci-sions in the continuous state space and action space.In this paper,a cooperative defense with DDPG via swarms of unmanned aerial vehicle(UAV)is developed and validated,which has shown promising practical value in the effect of defending.We solve the sparse rewards problem of reinforcement learning pair in a long-term task by building the reward function of UAV swarms and optimizing the learning process of artificial neural network based on the DDPG algorithm to reduce the vibration in the learning process.The experimental results show that the DDPG algorithm can guide the UAVs swarm to perform the defense task efficiently,meeting the requirements of a UAV swarm for non-centralization,autonomy,and promoting the intelligent development of UAVs swarm as well as the decision-making process.展开更多
针对光储直流微电网易受光伏资源波动、负荷侧波动等不确定扰动影响,进而引发的直流母线电压波动问题,在传统自抗扰控制(linear active disturbance rejection control,LADRC)的基础上,提出一种参数动态协同自抗扰控制(dynamic coordina...针对光储直流微电网易受光伏资源波动、负荷侧波动等不确定扰动影响,进而引发的直流母线电压波动问题,在传统自抗扰控制(linear active disturbance rejection control,LADRC)的基础上,提出一种参数动态协同自抗扰控制(dynamic coordination of parameters for active disturbance rejection control,DCLADRC),引入两个新的观测变量并增加一维带宽参数,旨在通过深度确定性策略梯度(deterministic policy gradient,DDPG)算法动态调整两级带宽间的协调因子k,提高观测器多频域扰动下的观测精度及收敛速度,优化控制器的抗扰性,增强母线电压稳定性,从而使得储能能够更好地发挥“削峰填谷”的调节作用。物理实验结果表明,受到扰动后,对比LADRC与双闭环比例积分(double closed loop proportion-integration,Double_PI)控制,所提的DCLADRC电压偏移量分别减少了75%和83%。展开更多
针对柔性直流输电系统(voltage source converter based high voltage direct current transmission,VSC-HVDC)控制参数设计过程中存在的鲁棒性差、依赖已知电路参数、工程设计经验化等问题,提出一种基于马尔科夫转换场(Markov transiti...针对柔性直流输电系统(voltage source converter based high voltage direct current transmission,VSC-HVDC)控制参数设计过程中存在的鲁棒性差、依赖已知电路参数、工程设计经验化等问题,提出一种基于马尔科夫转换场(Markov transition field,MTF)与深度确定性策略梯度算法(deep deterministic policy gradient,DDPG)结合的鲁棒性强、不依赖电路参数特性以及可视化的VSC-HVDC控制参数优化设计方法。首先,采用马尔科夫转换场将电路功率、电压等一维时序波形数据转换为二维马尔科夫转换场域图像并使用马尔科夫转换场损失函数(Markov transition field loss,MTFL)判断二维转换域图的数据波动性;其次,将MTFL损失函数与DDPG算法相结合,综合利用MTFL损失函数对系统输出时序数据动态特性评价能力更强的优点和DDPG算法泛化性能优秀的特点,实现VSC-HVDC系统控制参数优化;最后,通过MATLAB模拟和实验结果验证该方法的有效性。展开更多
In the complex and variable deep-sea environment,the compensation control of ship motion ensures the safety and efficiency of equipment installation and transportation in offshore wind farms.However,the ship motion po...In the complex and variable deep-sea environment,the compensation control of ship motion ensures the safety and efficiency of equipment installation and transportation in offshore wind farms.However,the ship motion posture compensation control system is severely affected by uncertainties,which significantly impact the accuracy of compensation control.In this paper,we propose a ship three-degree-of-freedom(3-DoF)motion posture stabilization control method based on the DTW-LSTM-MATD3 algorithm.We use the multi-agent twin delayed deep deterministic policy gradient(MATD3)to control a platform with six electric cylinders to achieve stable control.However,owing to random noise affecting the ship’s motion posture,we use a dynamic time warping(DTW)algorithm to distinguish between high-frequency noise and low-frequency tracking signals.Further,we embed a long short-term memory(LSTM)network into the MATD3 network to better align the Critic network’s training with the true Q-value.We use a combined reward function to enhance the agent’s exploration capability in complex dynamic environments.Finally,verification was conducted under sixth-level,abrupt sea conditions with high-frequency noise,as well as under real abrupt sea conditions,and a generalization test was also carried out.Simulation results show that the proposed DTW-LSTM-MATD3 method has great compensation control ability.展开更多
基金supported by the Key Research and Development Program of Shaanxi(2022GY-089)the Natural Science Basic Research Program of Shaanxi(2022JQ-593).
摘要The deep deterministic policy gradient(DDPG)algo-rithm is an off-policy method that combines two mainstream reinforcement learning methods based on value iteration and policy iteration.Using the DDPG algorithm,agents can explore and summarize the environment to achieve autonomous deci-sions in the continuous state space and action space.In this paper,a cooperative defense with DDPG via swarms of unmanned aerial vehicle(UAV)is developed and validated,which has shown promising practical value in the effect of defending.We solve the sparse rewards problem of reinforcement learning pair in a long-term task by building the reward function of UAV swarms and optimizing the learning process of artificial neural network based on the DDPG algorithm to reduce the vibration in the learning process.The experimental results show that the DDPG algorithm can guide the UAVs swarm to perform the defense task efficiently,meeting the requirements of a UAV swarm for non-centralization,autonomy,and promoting the intelligent development of UAVs swarm as well as the decision-making process.
摘要针对光储直流微电网易受光伏资源波动、负荷侧波动等不确定扰动影响,进而引发的直流母线电压波动问题,在传统自抗扰控制(linear active disturbance rejection control,LADRC)的基础上,提出一种参数动态协同自抗扰控制(dynamic coordination of parameters for active disturbance rejection control,DCLADRC),引入两个新的观测变量并增加一维带宽参数,旨在通过深度确定性策略梯度(deterministic policy gradient,DDPG)算法动态调整两级带宽间的协调因子k,提高观测器多频域扰动下的观测精度及收敛速度,优化控制器的抗扰性,增强母线电压稳定性,从而使得储能能够更好地发挥“削峰填谷”的调节作用。物理实验结果表明,受到扰动后,对比LADRC与双闭环比例积分(double closed loop proportion-integration,Double_PI)控制,所提的DCLADRC电压偏移量分别减少了75%和83%。
摘要针对柔性直流输电系统(voltage source converter based high voltage direct current transmission,VSC-HVDC)控制参数设计过程中存在的鲁棒性差、依赖已知电路参数、工程设计经验化等问题,提出一种基于马尔科夫转换场(Markov transition field,MTF)与深度确定性策略梯度算法(deep deterministic policy gradient,DDPG)结合的鲁棒性强、不依赖电路参数特性以及可视化的VSC-HVDC控制参数优化设计方法。首先,采用马尔科夫转换场将电路功率、电压等一维时序波形数据转换为二维马尔科夫转换场域图像并使用马尔科夫转换场损失函数(Markov transition field loss,MTFL)判断二维转换域图的数据波动性;其次,将MTFL损失函数与DDPG算法相结合,综合利用MTFL损失函数对系统输出时序数据动态特性评价能力更强的优点和DDPG算法泛化性能优秀的特点,实现VSC-HVDC系统控制参数优化;最后,通过MATLAB模拟和实验结果验证该方法的有效性。
基金supported by the National Natural Science Foundation of China(No.52105466).
摘要In the complex and variable deep-sea environment,the compensation control of ship motion ensures the safety and efficiency of equipment installation and transportation in offshore wind farms.However,the ship motion posture compensation control system is severely affected by uncertainties,which significantly impact the accuracy of compensation control.In this paper,we propose a ship three-degree-of-freedom(3-DoF)motion posture stabilization control method based on the DTW-LSTM-MATD3 algorithm.We use the multi-agent twin delayed deep deterministic policy gradient(MATD3)to control a platform with six electric cylinders to achieve stable control.However,owing to random noise affecting the ship’s motion posture,we use a dynamic time warping(DTW)algorithm to distinguish between high-frequency noise and low-frequency tracking signals.Further,we embed a long short-term memory(LSTM)network into the MATD3 network to better align the Critic network’s training with the true Q-value.We use a combined reward function to enhance the agent’s exploration capability in complex dynamic environments.Finally,verification was conducted under sixth-level,abrupt sea conditions with high-frequency noise,as well as under real abrupt sea conditions,and a generalization test was also carried out.Simulation results show that the proposed DTW-LSTM-MATD3 method has great compensation control ability.
摘要针对光储微电网中辐照度突变与负载投切引发的直流母线电压波动问题,为同步提升系统抗扰能力、动态响应速度及电流轨迹跟踪的精度。提出双环协同控制架构:电压外环采用优化策略梯度(policy gradient,PG)算法优化线性自抗扰控制(linear active disturbance rejection control,LADRC)的带宽参数,实现无需依赖精确系统模型的自适应参数整定;电流内环以有限控制集模型预测控制(finite control set-model predictive control,FCS-MPC)对目标函数集成轨迹柔性因子,量化电流偏差与变化率的综合性能,克服传统MPC单一性能优化的缺陷。基于超级电容与蓄电池的互补特性,对电压外环输出的参考电流滤波分频,生成差异化储能单元电流参考指令,优化储能系统功率分配效率。最后通过仿真对比传统PI(proportional-integral)、MPC、LADRC及协同控制策略,并以整体性能指标验证所提方法的优越性。