As a government-regulated public service,traffic signal control(TSC)requires reliable and transparent decision-making.However,existing deep reinforcement learning(DRL)methods,despite improvements in control accuracy,s...As a government-regulated public service,traffic signal control(TSC)requires reliable and transparent decision-making.However,existing deep reinforcement learning(DRL)methods,despite improvements in control accuracy,still lack explainability and generalisation,severely limiting their applicability in real-world environments.To address the challenges above,this paper proposes GenEx-TSC,a generalisable and explainable TSC method that integrates deep reinforcement learning with large language models(LLMs).First,starting from vehicle-level states,we train a DRL agent incorporating intersection physical heterogeneity and neighbourhood information,which lays the evaluation foundation for constructing a high-quality LLM dataset.Subsequently,the LLM agent is optimised through a two-stage training mechanism.In the distillation stage,a lightweight LLM agent is trained using the reasoning trajectories of a larger-scale LLM agent,inheriting its semantic understanding and decision-generation capabilities and in the alignment stage,the DRL evaluation network is employed to calibrate the outputs of the distilled LLM agent,ensuring that the generated cycle-level signal timing strategies are both efficient and interpretable.We synthesise 10 intersection networks with different physical attributes in SUMO and set traffic flows of varying scales.Experimental results across diverse traffic environments demonstrate that the proposed GenEx-TSC exhibits clear advantages over traditional methods,mainstream DRL methods and LLM baselines in terms of control accuracy,generalisation and explainability.展开更多
Unmanned Aerial Vehicle(UAV)plays a prominent role in various fields,and autonomous navigation is a crucial component of UAV intelligence.Deep Reinforcement Learning(DRL)has expanded the research avenues for addressin...Unmanned Aerial Vehicle(UAV)plays a prominent role in various fields,and autonomous navigation is a crucial component of UAV intelligence.Deep Reinforcement Learning(DRL)has expanded the research avenues for addressing challenges in autonomous navigation.Nonetheless,challenges persist,including getting stuck in local optima,consuming excessive computations during action space exploration,and neglecting deterministic experience.This paper proposes a noise-driven enhancement strategy.In accordance with the overall learning phases,a global noise control method is designed,while a differentiated local noise control method is developed by analyzing the exploration demands of four typical situations encountered by UAV during navigation.Both methods are integrated into a dual-model for noise control to regulate action space exploration.Furthermore,noise dual experience replay buffers are designed to optimize the rational utilization of both deterministic and noisy experience.In uncertain environments,based on the Twin Delay Deep Deterministic Policy Gradient(TD3)algorithm with Long Short-Term Memory(LSTM)network and Priority Experience Replay(PER),a Noise-Driven Enhancement Priority Memory TD3(NDE-PMTD3)is developed.We established a simulation environment to compare different algorithms,and the performance of the algorithms is analyzed in various scenarios.The training results indicate that the proposed algorithm accelerates the convergence speed and enhances the convergence stability.In test experiments,the proposed algorithm successfully and efficiently performs autonomous navigation tasks in diverse environments,demonstrating superior generalization results.展开更多
Vehicle Edge Computing(VEC)and Cloud Computing(CC)significantly enhance the processing efficiency of delay-sensitive and computation-intensive applications by offloading compute-intensive tasks from resource-constrain...Vehicle Edge Computing(VEC)and Cloud Computing(CC)significantly enhance the processing efficiency of delay-sensitive and computation-intensive applications by offloading compute-intensive tasks from resource-constrained onboard devices to nearby Roadside Unit(RSU),thereby achieving lower delay and energy consumption.However,due to the limited storage capacity and energy budget of RSUs,it is challenging to meet the demands of the highly dynamic Internet of Vehicles(IoV)environment.Therefore,determining reasonable service caching and computation offloading strategies is crucial.To address this,this paper proposes a joint service caching scheme for cloud-edge collaborative IoV computation offloading.By modeling the dynamic optimization problem using Markov Decision Processes(MDP),the scheme jointly optimizes task delay,energy consumption,load balancing,and privacy entropy to achieve better quality of service.Additionally,a dynamic adaptive multi-objective deep reinforcement learning algorithm is proposed.Each Double Deep Q-Network(DDQN)agent obtains rewards for different objectives based on distinct reward functions and dynamically updates the objective weights by learning the value changes between objectives using Radial Basis Function Networks(RBFN),thereby efficiently approximating the Pareto-optimal decisions for multiple objectives.Extensive experiments demonstrate that the proposed algorithm can better coordinate the three-tier computing resources of cloud,edge,and vehicles.Compared to existing algorithms,the proposed method reduces task delay and energy consumption by 10.64%and 5.1%,respectively.展开更多
With the advent of sixth-generation mobile communications(6G),space-air-ground integrated networks have become mainstream.This paper focuses on collaborative scheduling for mobile edge computing(MEC)under a three-tier...With the advent of sixth-generation mobile communications(6G),space-air-ground integrated networks have become mainstream.This paper focuses on collaborative scheduling for mobile edge computing(MEC)under a three-tier heterogeneous architecture composed of mobile devices,unmanned aerial vehicles(UAVs),and macro base stations(BSs).This scenario typically faces fast channel fading,dynamic computational loads,and energy constraints,whereas classical queuing-theoretic or convex-optimization approaches struggle to yield robust solutions in highly dynamic settings.To address this issue,we formulate a multi-agent Markov decision process(MDP)for an air-ground-fused MEC system,unify link selection,bandwidth/power allocation,and task offloading into a continuous action space and propose a joint scheduling strategy that is based on an improved MATD3 algorithm.The improvements include Alternating Layer Normalization(ALN)in the actor to suppress gradient variance,Residual Orthogonalization(RO)in the critic to reduce the correlation between the twin Q-value estimates,and a dynamic-temperature reward to enable adaptive trade-offs during training.On a multi-user,dual-link simulation platform,we conduct ablation and baseline comparisons.The results reveal that the proposed method has better convergence and stability.Compared with MADDPG,TD3,and DSAC,our algorithm achieves more robust performance across key metrics.展开更多
Effective partitioning is crucial for enabling parallel restoration of power systems after blackouts.This paper proposes a novel partitioning method based on deep reinforcement learning.First,the partitioning decision...Effective partitioning is crucial for enabling parallel restoration of power systems after blackouts.This paper proposes a novel partitioning method based on deep reinforcement learning.First,the partitioning decision process is formulated as a Markov decision process(MDP)model to maximize the modularity.Corresponding key partitioning constraints on parallel restoration are considered.Second,based on the partitioning objective and constraints,the reward function of the partitioning MDP model is set by adopting a relative deviation normalization scheme to reduce mutual interference between the reward and penalty in the reward function.The soft bonus scaling mechanism is introduced to mitigate overestimation caused by abrupt jumps in the reward.Then,the deep Q network method is applied to solve the partitioning MDP model and generate partitioning schemes.Two experience replay buffers are employed to speed up the training process of the method.Finally,case studies on the IEEE 39-bus test system demonstrate that the proposed method can generate a high-modularity partitioning result that meets all key partitioning constraints,thereby improving the parallelism and reliability of the restoration process.Moreover,simulation results demonstrate that an appropriate discount factor is crucial for ensuring both the convergence speed and the stability of the partitioning training.展开更多
The published article titled“A Lightweight Multimodal Deep Fusion Network for Face Antis Poofing with Cross-Axial Attention and Deep Reinforcement Learning Technique”has been retracted from Computers,Materials&C...The published article titled“A Lightweight Multimodal Deep Fusion Network for Face Antis Poofing with Cross-Axial Attention and Deep Reinforcement Learning Technique”has been retracted from Computers,Materials&Continua,Vol.85,No.3,2025,pp.5671-5702.展开更多
Deep Reinforcement Learning(DRL)offers a powerful,model-free,and data-driven approach for the navigation and control of Autonomous Surface Vessels(ASVs).The primary challenge,however,lies in the extensive training req...Deep Reinforcement Learning(DRL)offers a powerful,model-free,and data-driven approach for the navigation and control of Autonomous Surface Vessels(ASVs).The primary challenge,however,lies in the extensive training required for an agent to converge to an effective policy within a complex simulation,leading to significant computational overhead.This paper presents a multi-stage training framework that uses Transfer Learning to pass knowledge between different simulation models,resulting in a highly robust DRL controller for ASVs.The proposed framework utilizes the Deep Deterministic Policy Gradient(DDPG)algorithm to develop the data-driven controller.First,a foundational policy is efficiently learned using a simplified first-order Nomoto dynamics and second-order Nomoto dynamics,which captures the fundamental vessel dynamics.This pre-trained policy is then transferred to a complex,nonlinear Manoeuvring Modelling Group(MMG)model,significantly accelerating training convergence.Subsequently,the agent is fine-tuned within the MMG simulation with environmental disturbances.The models are evaluated on various trajectories during testing to ensure robust performance.The accuracy of the DRL controller is assessed by measuring heading error(eψ)and cross-track error(ye).A traditional Proportional-Integral-Derivative(PID)controller is implemented and compared to benchmark the DRL controller's effectiveness,to highlight the relative advantages and limitations of each approach.展开更多
Previous studies have demonstrated that the immunoglobulin G(IgG)N-glycome and transcriptome are potential biochemical signatures of chronological and biological ages,and several aging clocks have been developed.By in...Previous studies have demonstrated that the immunoglobulin G(IgG)N-glycome and transcriptome are potential biochemical signatures of chronological and biological ages,and several aging clocks have been developed.By integrating the IgG N-glycome and transcriptome,we propose a novel aging clock,gtAge.We developed a deep reinforcement learning-based multiomics integration method called AlphaSnake.The results showed that AlphaSnake achieved a predicted coefficient of determination(R2)value of0.853,outperforming the concatenation-based integration method(R2=0.820)The gtAge estimated by AlphaSnake explained up to 85.3%of the variance in chronological age,which was higher than that in age predicted from IgG N-glycome solely(gAge;R2=0.290)and age predicted from transcriptome solely(tAge;R2=0.812).We also found that the delta age-the difference between the predicted age and chronological age-was associated with several age-related phenotypes.Both delta gtAge and tAge were negatively associated with high-density lipoprotein(p=0.02 and p=0.022,respectively),whereas delta gAge was positively correlated with cholesterol(p=0.006),triglyceride(p=0.002),fasting plasma glucose(p=0.014),low-density lipoprotein(p=0.006),and glycated hemoglobin(p=0.039).These findings suggest that gtAge,tAge,and gAge are potential biomarkers for biological age.展开更多
Real-time onboard performance modeling for aero-engines holds a pivotal position in engine control and fault diagnosis.Nevertheless,the relentless pursuit of enhancing aero-engine performance has led to increasingly i...Real-time onboard performance modeling for aero-engines holds a pivotal position in engine control and fault diagnosis.Nevertheless,the relentless pursuit of enhancing aero-engine performance has led to increasingly intricate engine architectures,posing significant challenges to the real-time execution of onboard models.To improve the real-time performance while maintaining accuracy,and address the inherent problem that traditional models can only start to simulate at a fixed point,an aero-engine hybrid onboard model based on deep reinforcement learning is proposed.This study focuses on the high-flow dual variable cycle engine,firstly delving into the engine's thermodynamic principles comprehensively,then establishing and validating the Component Level Model(CLM),which serve as the inner loop of hybrid model.Subsequently,Twin Delayed Deep Deterministic(TD3)policy gradient algorithm is integrated to establish the outer loop of hybrid model within full flight envelope and operating states.Simulation results show that the hybrid model has the characteristics of faster convergence,higher accuracy and superior realtime performance.Finally,the inner-outer loop design enables hybrid onboard model to approach the domain of solution quickly,significantly reducing the calculation time of simulation initialization and ensuring model convergence throughout transition processes.展开更多
The increasing integration of cyber-physical components in Industry 4.0 water infrastructures has heightened the risk of false data injection(FDI)attacks,posing critical threats to operational integrity,resource manag...The increasing integration of cyber-physical components in Industry 4.0 water infrastructures has heightened the risk of false data injection(FDI)attacks,posing critical threats to operational integrity,resource management,and public safety.Traditional detection mechanisms often struggle to generalize across heterogeneous environments or adapt to sophisticated,stealthy threats.To address these challenges,we propose a novel evolutionary optimized transformer-based deep reinforcement learning framework(Evo-Transformer-DRL)designed for robust and adaptive FDI detection in smart water infrastructures.The proposed architecture integrates three powerful paradigms:a transformer encoder for modeling complex temporal dependencies in multivariate time series,a DRL agent for learning optimal decision policies in dynamic environments,and an evolutionary optimizer to fine-tune model hyper-parameters.This synergy enhances detection performance while maintaining adaptability across varying data distributions.Specifically,hyper-parameters of both the transformer and DRL modules are optimized using an improved grey wolf optimizer(IGWO),ensuring a balanced trade-off between detection accuracy and computational efficiency.The model is trained and evaluated on three realistic Industry 4.0 water datasets:secure water treatment(SWaT),water distribution(WADI),and battle of the attack detection algorithms(BATADAL),which capture diverse attack scenarios in smart treatment and distribution systems.Comparative analysis against state-of-the-art baselines including Transformer,DRL,bidirectional encoder representations from transformers(BERT),convolutional neural network(CNN),long short-term memory(LSTM),and support vector machines(SVM)demonstrates that our proposed Evo-Transformer-DRL framework consistently outperforms others in key metrics such as accuracy,recall,area under the curve(AUC),and execution time.Notably,it achieves a maximum detection accuracy of 99.19%,highlighting its strong generalization capability across different testbeds.These results confirm the suitability of our hybrid framework for real-world Industry 4.0 deployment,where rapid adaptation,scalability,and reliability are paramount for securing critical infrastructure systems.展开更多
As joint operations have become a key trend in modern military development,unmanned aerial vehicles(UAVs)play an increasingly important role in enhancing the intelligence and responsiveness of combat systems.However,t...As joint operations have become a key trend in modern military development,unmanned aerial vehicles(UAVs)play an increasingly important role in enhancing the intelligence and responsiveness of combat systems.However,the heterogeneity of aircraft,partial observability,and dynamic uncertainty in operational airspace pose significant challenges to autonomous collision avoidance using traditional methods.To address these issues,this paper proposes an adaptive collision avoidance approach for UAVs based on deep reinforcement learning.First,a unified uncertainty model incorporating dynamic wind fields is constructed to capture the complexity of joint operational environments.Then,to effectively handle the heterogeneity between manned and unmanned aircraft and the limitations of dynamic observations,a sector-based partial observation mechanism is designed.A Dynamic Threat Prioritization Assessment algorithm is also proposed to evaluate potential collision threats from multiple dimensions,including time to closest approach,minimum separation distance,and aircraft type.Furthermore,a Hierarchical Prioritized Experience Replay(HPER)mechanism is introduced,which classifies experience samples into high,medium,and low priority levels to preferentially sample critical experiences,thereby improving learning efficiency and accelerating policy convergence.Simulation results show that the proposed HPER-D3QN algorithm outperforms existing methods in terms of learning speed,environmental adaptability,and robustness,significantly enhancing collision avoidance performance and convergence rate.Finally,transfer experiments on a high-fidelity battlefield airspace simulation platform validate the proposed method's deployment potential and practical applicability in complex,real-world joint operational scenarios.展开更多
Assessing the vulnerability of complex systems requires effective hypergraph dismantling strategies,yet existing methods struggle with the dynamic nature of cascading failures and the rugged optimization landscapes of...Assessing the vulnerability of complex systems requires effective hypergraph dismantling strategies,yet existing methods struggle with the dynamic nature of cascading failures and the rugged optimization landscapes of high-order networks.In this paper,we propose a novel framework:hypergraph dismantling via evolutionary deep reinforcement learning(HD-EDR).First,we model a realistic dismantling environment incorporating hyperdegree-based and residual-capacitybased load redistribution mechanisms.Second,we introduce a hybrid learning architecture that synergizes the global exploration of evolutionary strategies with the gradient-based exploitation of deep reinforcement learning.A bidirectional parameter synchronization mechanism is designed to prevent the agent from being trapped in local optima.Furthermore,we integrate an inductive encoder to capture the evolving high-order dependencies of the residual network in real time.Extensive experiments across nine real-world datasets demonstrate that our framework significantly outperforms state-of-the-art baselines,providing a highly effective and robust strategy for maximizing structural damage in high-order networks.展开更多
To address the high costs and operational instability of distribution networks caused by the large-scale integration of distributed energy resources(DERs)(such as photovoltaic(PV)systems,wind turbines(WT),and energy s...To address the high costs and operational instability of distribution networks caused by the large-scale integration of distributed energy resources(DERs)(such as photovoltaic(PV)systems,wind turbines(WT),and energy storage(ES)devices),and the increased grid load fluctuations and safety risks due to uncoordinated electric vehicles(EVs)charging,this paper proposes a novel dual-scale hierarchical collaborative optimization strategy.This strategy decouples system-level economic dispatch from distributed EV agent control,effectively solving the resource coordination conflicts arising from the high computational complexity,poor scalability of existing centralized optimization,or the reliance on local information decision-making in fully decentralized frameworks.At the lower level,an EV charging and discharging model with a hybrid discrete-continuous action space is established,and optimized using an improved Parameterized Deep Q-Network(PDQN)algorithm,which directly handles mode selection and power regulation while embedding physical constraints to ensure safety.At the upper level,microgrid(MG)operators adopt a dynamic pricing strategy optimized through Deep Reinforcement Learning(DRL)to maximize economic benefits and achieve peak-valley shaving.Simulation results show that the proposed strategy outperforms traditional methods,reducing the total operating cost of the MG by 21.6%,decreasing the peak-to-valley load difference by 33.7%,reducing the number of voltage limit violations by 88.9%,and lowering the average electricity cost for EV users by 15.2%.This method brings a win-win result for operators and users,providing a reliable and efficient scheduling solution for distribution networks with high renewable energy penetration rates.展开更多
In response to the rising demand for low-latency,computation-intensive applications in vehicular networks,this paper proposes an adaptive task offloading approach for Vehicle-to-Everything(V2X)environments.Leveraging ...In response to the rising demand for low-latency,computation-intensive applications in vehicular networks,this paper proposes an adaptive task offloading approach for Vehicle-to-Everything(V2X)environments.Leveraging an enhanced Multi-Agent Deep Deterministic Policy Gradient(MADDPG)algorithm with an attention mechanism,the proposed approach optimizes computation offloading and resource allocation,aiming to minimize energy consumption and service delay.In this paper,vehicles dynamically offload computing-intensive tasks to both nearby vehicles through V2V links and roadside units through V2I links.The adaptive attention mechanism enables the system to prioritize relevant state information,leading to faster convergence.Simulations conducted in a realistic urban V2X scenario demonstrate that the proposed Attention-enhanced MADDPG(AT-MADDPG)algorithm significantly improves performance,achieving notable reductions in both energy consumption and latency compared to baseline algorithms,especially in high-demand,dynamic scenarios.展开更多
Mobile Edge Computing(MEC)facilitates the rapid response and energy-efficient execution of tasks on mobile devices.However,determining whether and where to offload tasks remains a significant challenge due to the cons...Mobile Edge Computing(MEC)facilitates the rapid response and energy-efficient execution of tasks on mobile devices.However,determining whether and where to offload tasks remains a significant challenge due to the constantly changing character of workloads in MEC environments.To address this issue,this paper proposes PreAlloc-A2C—a deep reinforcement learning actor-critic-based framework that calculates allocation scores by leveraging both task features(task size,required completion time,and waiting time)and server features(queue length and historical workload).This design enables fully distributed task offloading decisions without centralized coordination.Additionally,a Long Short-Term Memory(LSTM)network is integrated to forecast impending server loads,thereby supporting adaptive scheduling.A tailored reward function is also designed to jointly optimize three key performance metrics:task delay,device energy consumption,and task drop rate.Extensive experiments are conducted to evaluate PreAlloc-A2C against five baseline algorithms:Particle Swarm Optimization(PSO),Advantage Actor-Critic(A2C),Deep Q-Network(DQN),Double Deep Q-Network(DDQN),and Dueling Deep Q-Network(Dueling DQN).The results show that PreAlloc-A2C outperforms all baselines,achieving lower latency,reduced energy consumption,and a lower task drop rate.展开更多
Underwater images frequently suffer from chromatic distortion,blurred details,and low contrast,posing significant challenges for enhancement.This paper introduces AquaTree,a novel underwater image enhancement(UIE)meth...Underwater images frequently suffer from chromatic distortion,blurred details,and low contrast,posing significant challenges for enhancement.This paper introduces AquaTree,a novel underwater image enhancement(UIE)method that reformulates the task as a Markov Decision Process(MDP)through the integration of Monte Carlo Tree Search(MCTS)and deep reinforcement learning(DRL).The framework employs an action space of 25 enhancement operators,strategically grouped for basic attribute adjustment,color component balance,correction,and deblurring.Exploration within MCTS is guided by a dual-branch convolutional network,enabling intelligent sequential operator selection.Our core contributions include:(1)a multimodal state representation combining CIELab color histograms with deep perceptual features,(2)a dual-objective reward mechanism optimizing chromatic fidelity and perceptual consistency,and(3)an alternating training strategy co-optimizing enhancement sequences and network parameters.We further propose two inference schemes:an MCTS-based approach prioritizing accuracy at higher computational cost,and an efficient network policy enabling real-time processing with minimal quality loss.Comprehensive evaluations on the UIEB Dataset and Color correction and haze removal comparisons on the U45 Dataset demonstrate AquaTree’s superiority,significantly outperforming nine state-of-the-art methods across five established underwater image quality metrics.展开更多
As the types of traffic requests increase,the elastic optical network(EON)is considered as a promising architecture to carry multiple types of traffic requests simultaneously,including immediate reservation(IR)and adv...As the types of traffic requests increase,the elastic optical network(EON)is considered as a promising architecture to carry multiple types of traffic requests simultaneously,including immediate reservation(IR)and advance reservation(AR).Various resource allocation schemes for IR/AR requests have been designed in EON to reduce bandwidth blocking probability(BBP).However,these schemes do not consider different transmission requirements of IR requests and cannot maintain a low BBP for high-priority requests.In this paper,multi-priority is considered in the hybrid IR/AR request scenario.We modify the asynchronous advantage actor critic(A3C)model and propose an A3C-assisted priority resource allocation(APRA)algorithm.The APRA integrates priority and transmission quality of IR requests to design the A3C reward function,then dynamically allocates dedicated resources for different IR requests according to the time-varying requirements.By maximizing the reward,the transmission quality of IR requests can be matched with the priority,and lower BBP for high-priority IR requests can be ensured.Simulation results show that the APRA reduces the BBP of high-priority IR requests from 0.0341 to0.0138,and the overall network operation gain is improved by 883 compared to the scheme without considering the priority.展开更多
At present,energy consumption is one of the main bottlenecks in autonomous mobile robot development.To address the challenge of high energy consumption in path planning for autonomous mobile robots navigating unknown ...At present,energy consumption is one of the main bottlenecks in autonomous mobile robot development.To address the challenge of high energy consumption in path planning for autonomous mobile robots navigating unknown and complex environments,this paper proposes an Attention-Enhanced Dueling Deep Q-Network(ADDueling DQN),which integrates a multi-head attention mechanism and a prioritized experience replay strategy into a Dueling-DQN reinforcement learning framework.A multi-objective reward function,centered on energy efficiency,is designed to comprehensively consider path length,terrain slope,motion smoothness,and obstacle avoidance,enabling optimal low-energy trajectory generation in 3D space from the source.The incorporation of a multihead attention mechanism allows the model to dynamically focus on energy-critical state features—such as slope gradients and obstacle density—thereby significantly improving its ability to recognize and avoid energy-intensive paths.Additionally,the prioritized experience replay mechanism accelerates learning from key decision-making experiences,suppressing inefficient exploration and guiding the policy toward low-energy solutions more rapidly.The effectiveness of the proposed path planning algorithm is validated through simulation experiments conducted in multiple off-road scenarios.Results demonstrate that AD-Dueling DQN consistently achieves the lowest average energy consumption across all tested environments.Moreover,the proposed method exhibits faster convergence and greater training stability compared to baseline algorithms,highlighting its global optimization capability under energy-aware objectives in complex terrains.This study offers an efficient and scalable intelligent control strategy for the development of energy-conscious autonomous navigation systems.展开更多
This paper investigates the joint resource allocation problem in Reconfigurable Intelligent Surface(RIS)-assisted cooperative non-orthogonal multiple access device-to-device(CNOMA-D2D)cellular networks.To tackle the h...This paper investigates the joint resource allocation problem in Reconfigurable Intelligent Surface(RIS)-assisted cooperative non-orthogonal multiple access device-to-device(CNOMA-D2D)cellular networks.To tackle the high-dimensional non-convex joint optimization of power control,RIS phase configuration and channel assignment,we propose an integrated user pairing strategy,PIP-UP,quantifying utility through factors,phase alignment,interference suppression and power difference,neglected in existing methods.Furthermore,we develop a hybrid deep reinforcement learning algorithm,A3TD,combining the parallel exploration capability of Asynchronous Advantage Actor-Critic(A3C)with the stable continuous optimization of Twin Delayed Deep Deterministic Policy Gradient(TD3).This integration enables efficient and robust joint optimization of D2D channel allocation,transmit power,and RIS phase shifts.Simulation results demonstrate that the proposed A3TD algorithm significantly outperforms baseline algorithms,Actor-Critic(AC),Deep Deterministic Policy Gradient(DDPG)and TD3,in terms of sum rate and convergence speed,validating its effectiveness for resource management in complex RIS-assisted CNOMA-D2D networks.展开更多
Deep reinforcement learning(DRL)has become an important method in Unmanned Aerial Vehicle(UAV)path planning,but the field still lacks a dedicated bibliometric review that summarizes its publication patterns,intellectu...Deep reinforcement learning(DRL)has become an important method in Unmanned Aerial Vehicle(UAV)path planning,but the field still lacks a dedicated bibliometric review that summarizes its publication patterns,intellectual structure,and thematic evolution.This study analyzes 1402 Web of Science publications from 2010 to 2025 using CiteSpace,VOSviewer,and the Bibliometrix R package.Three main findings are reported.First,the bibliometric evidence suggests a four-phase evolution of the field—foundational exploration(2015–2016),continuous-control breakthrough(2017–2019),multi-agent collaborative coordination(2020–2022),and complexscenario integration(2023–2025)—as reflected in publication trends,keyword bursts,and co-citation clusters.Second,co-citation and keyword analyses indicate a gradual shift from geometric navigation toward the joint consideration of communication,energy,and mission objectives,a pattern also reflected in the prominence of Internet of Things(IoT)and vehicular-technology journals.Third,burst and clustering results highlight several active research directions,including multi-objective cooperative decision-making,hierarchical planning architectures that combine global perception with local control,and communication–energy co-design.Rather than offering prescriptive conclusions,these results provide descriptive bibliometric evidence that may help researchers understand the development and emerging priorities of DRL-based UAV path planning.展开更多
基金the National Natural Science Foundation of China under(Grant No.62501094)in part by the Natural Science Foundation of Chongqing under(Grant Nos.CSTB2025NSCQLZX0152,CSTB2024NSCQ-LZX0134 and CSTB2025NSCQ-LZX0052).
摘要As a government-regulated public service,traffic signal control(TSC)requires reliable and transparent decision-making.However,existing deep reinforcement learning(DRL)methods,despite improvements in control accuracy,still lack explainability and generalisation,severely limiting their applicability in real-world environments.To address the challenges above,this paper proposes GenEx-TSC,a generalisable and explainable TSC method that integrates deep reinforcement learning with large language models(LLMs).First,starting from vehicle-level states,we train a DRL agent incorporating intersection physical heterogeneity and neighbourhood information,which lays the evaluation foundation for constructing a high-quality LLM dataset.Subsequently,the LLM agent is optimised through a two-stage training mechanism.In the distillation stage,a lightweight LLM agent is trained using the reasoning trajectories of a larger-scale LLM agent,inheriting its semantic understanding and decision-generation capabilities and in the alignment stage,the DRL evaluation network is employed to calibrate the outputs of the distilled LLM agent,ensuring that the generated cycle-level signal timing strategies are both efficient and interpretable.We synthesise 10 intersection networks with different physical attributes in SUMO and set traffic flows of varying scales.Experimental results across diverse traffic environments demonstrate that the proposed GenEx-TSC exhibits clear advantages over traditional methods,mainstream DRL methods and LLM baselines in terms of control accuracy,generalisation and explainability.
基金the Collaborative Innovation Project of Shanghai,China for the financial support。
摘要Unmanned Aerial Vehicle(UAV)plays a prominent role in various fields,and autonomous navigation is a crucial component of UAV intelligence.Deep Reinforcement Learning(DRL)has expanded the research avenues for addressing challenges in autonomous navigation.Nonetheless,challenges persist,including getting stuck in local optima,consuming excessive computations during action space exploration,and neglecting deterministic experience.This paper proposes a noise-driven enhancement strategy.In accordance with the overall learning phases,a global noise control method is designed,while a differentiated local noise control method is developed by analyzing the exploration demands of four typical situations encountered by UAV during navigation.Both methods are integrated into a dual-model for noise control to regulate action space exploration.Furthermore,noise dual experience replay buffers are designed to optimize the rational utilization of both deterministic and noisy experience.In uncertain environments,based on the Twin Delay Deep Deterministic Policy Gradient(TD3)algorithm with Long Short-Term Memory(LSTM)network and Priority Experience Replay(PER),a Noise-Driven Enhancement Priority Memory TD3(NDE-PMTD3)is developed.We established a simulation environment to compare different algorithms,and the performance of the algorithms is analyzed in various scenarios.The training results indicate that the proposed algorithm accelerates the convergence speed and enhances the convergence stability.In test experiments,the proposed algorithm successfully and efficiently performs autonomous navigation tasks in diverse environments,demonstrating superior generalization results.
基金supported by Key Science and Technology Program of Henan Province,China(Grant Nos.242102210147,242102210027)Fujian Province Young and Middle aged Teacher Education Research Project(Science and Technology Category)(No.JZ240101)(Corresponding author:Dong Yuan).
摘要Vehicle Edge Computing(VEC)and Cloud Computing(CC)significantly enhance the processing efficiency of delay-sensitive and computation-intensive applications by offloading compute-intensive tasks from resource-constrained onboard devices to nearby Roadside Unit(RSU),thereby achieving lower delay and energy consumption.However,due to the limited storage capacity and energy budget of RSUs,it is challenging to meet the demands of the highly dynamic Internet of Vehicles(IoV)environment.Therefore,determining reasonable service caching and computation offloading strategies is crucial.To address this,this paper proposes a joint service caching scheme for cloud-edge collaborative IoV computation offloading.By modeling the dynamic optimization problem using Markov Decision Processes(MDP),the scheme jointly optimizes task delay,energy consumption,load balancing,and privacy entropy to achieve better quality of service.Additionally,a dynamic adaptive multi-objective deep reinforcement learning algorithm is proposed.Each Double Deep Q-Network(DDQN)agent obtains rewards for different objectives based on distinct reward functions and dynamically updates the objective weights by learning the value changes between objectives using Radial Basis Function Networks(RBFN),thereby efficiently approximating the Pareto-optimal decisions for multiple objectives.Extensive experiments demonstrate that the proposed algorithm can better coordinate the three-tier computing resources of cloud,edge,and vehicles.Compared to existing algorithms,the proposed method reduces task delay and energy consumption by 10.64%and 5.1%,respectively.
摘要With the advent of sixth-generation mobile communications(6G),space-air-ground integrated networks have become mainstream.This paper focuses on collaborative scheduling for mobile edge computing(MEC)under a three-tier heterogeneous architecture composed of mobile devices,unmanned aerial vehicles(UAVs),and macro base stations(BSs).This scenario typically faces fast channel fading,dynamic computational loads,and energy constraints,whereas classical queuing-theoretic or convex-optimization approaches struggle to yield robust solutions in highly dynamic settings.To address this issue,we formulate a multi-agent Markov decision process(MDP)for an air-ground-fused MEC system,unify link selection,bandwidth/power allocation,and task offloading into a continuous action space and propose a joint scheduling strategy that is based on an improved MATD3 algorithm.The improvements include Alternating Layer Normalization(ALN)in the actor to suppress gradient variance,Residual Orthogonalization(RO)in the critic to reduce the correlation between the twin Q-value estimates,and a dynamic-temperature reward to enable adaptive trade-offs during training.On a multi-user,dual-link simulation platform,we conduct ablation and baseline comparisons.The results reveal that the proposed method has better convergence and stability.Compared with MADDPG,TD3,and DSAC,our algorithm achieves more robust performance across key metrics.
基金funded by the Beijing Engineering Research Center of Electric Rail Transportation.
摘要Effective partitioning is crucial for enabling parallel restoration of power systems after blackouts.This paper proposes a novel partitioning method based on deep reinforcement learning.First,the partitioning decision process is formulated as a Markov decision process(MDP)model to maximize the modularity.Corresponding key partitioning constraints on parallel restoration are considered.Second,based on the partitioning objective and constraints,the reward function of the partitioning MDP model is set by adopting a relative deviation normalization scheme to reduce mutual interference between the reward and penalty in the reward function.The soft bonus scaling mechanism is introduced to mitigate overestimation caused by abrupt jumps in the reward.Then,the deep Q network method is applied to solve the partitioning MDP model and generate partitioning schemes.Two experience replay buffers are employed to speed up the training process of the method.Finally,case studies on the IEEE 39-bus test system demonstrate that the proposed method can generate a high-modularity partitioning result that meets all key partitioning constraints,thereby improving the parallelism and reliability of the restoration process.Moreover,simulation results demonstrate that an appropriate discount factor is crucial for ensuring both the convergence speed and the stability of the partitioning training.
摘要The published article titled“A Lightweight Multimodal Deep Fusion Network for Face Antis Poofing with Cross-Axial Attention and Deep Reinforcement Learning Technique”has been retracted from Computers,Materials&Continua,Vol.85,No.3,2025,pp.5671-5702.
摘要Deep Reinforcement Learning(DRL)offers a powerful,model-free,and data-driven approach for the navigation and control of Autonomous Surface Vessels(ASVs).The primary challenge,however,lies in the extensive training required for an agent to converge to an effective policy within a complex simulation,leading to significant computational overhead.This paper presents a multi-stage training framework that uses Transfer Learning to pass knowledge between different simulation models,resulting in a highly robust DRL controller for ASVs.The proposed framework utilizes the Deep Deterministic Policy Gradient(DDPG)algorithm to develop the data-driven controller.First,a foundational policy is efficiently learned using a simplified first-order Nomoto dynamics and second-order Nomoto dynamics,which captures the fundamental vessel dynamics.This pre-trained policy is then transferred to a complex,nonlinear Manoeuvring Modelling Group(MMG)model,significantly accelerating training convergence.Subsequently,the agent is fine-tuned within the MMG simulation with environmental disturbances.The models are evaluated on various trajectories during testing to ensure robust performance.The accuracy of the DRL controller is assessed by measuring heading error(eψ)and cross-track error(ye).A traditional Proportional-Integral-Derivative(PID)controller is implemented and compared to benchmark the DRL controller's effectiveness,to highlight the relative advantages and limitations of each approach.
基金funded by an Australia–China International Collaborative Grant(NHMRC APP1112767-NSFC 81561128020)the European Union’s Horizon 2020 Research and Innovation Program under grant agreement(779238)+2 种基金the Edith Cowan University Higher Degree by Research Scholarship(ECU-HDR 10492768)the Western Australian Future Health Research and Innovation Funds(WANMA/EL2023-24/2 and WANMA/Ideas2024-25/5)the Edith Cowan University Early-Mid Career Researcher Grant Scheme(G1006465)。
摘要Previous studies have demonstrated that the immunoglobulin G(IgG)N-glycome and transcriptome are potential biochemical signatures of chronological and biological ages,and several aging clocks have been developed.By integrating the IgG N-glycome and transcriptome,we propose a novel aging clock,gtAge.We developed a deep reinforcement learning-based multiomics integration method called AlphaSnake.The results showed that AlphaSnake achieved a predicted coefficient of determination(R2)value of0.853,outperforming the concatenation-based integration method(R2=0.820)The gtAge estimated by AlphaSnake explained up to 85.3%of the variance in chronological age,which was higher than that in age predicted from IgG N-glycome solely(gAge;R2=0.290)and age predicted from transcriptome solely(tAge;R2=0.812).We also found that the delta age-the difference between the predicted age and chronological age-was associated with several age-related phenotypes.Both delta gtAge and tAge were negatively associated with high-density lipoprotein(p=0.02 and p=0.022,respectively),whereas delta gAge was positively correlated with cholesterol(p=0.006),triglyceride(p=0.002),fasting plasma glucose(p=0.014),low-density lipoprotein(p=0.006),and glycated hemoglobin(p=0.039).These findings suggest that gtAge,tAge,and gAge are potential biomarkers for biological age.
摘要Real-time onboard performance modeling for aero-engines holds a pivotal position in engine control and fault diagnosis.Nevertheless,the relentless pursuit of enhancing aero-engine performance has led to increasingly intricate engine architectures,posing significant challenges to the real-time execution of onboard models.To improve the real-time performance while maintaining accuracy,and address the inherent problem that traditional models can only start to simulate at a fixed point,an aero-engine hybrid onboard model based on deep reinforcement learning is proposed.This study focuses on the high-flow dual variable cycle engine,firstly delving into the engine's thermodynamic principles comprehensively,then establishing and validating the Component Level Model(CLM),which serve as the inner loop of hybrid model.Subsequently,Twin Delayed Deep Deterministic(TD3)policy gradient algorithm is integrated to establish the outer loop of hybrid model within full flight envelope and operating states.Simulation results show that the hybrid model has the characteristics of faster convergence,higher accuracy and superior realtime performance.Finally,the inner-outer loop design enables hybrid onboard model to approach the domain of solution quickly,significantly reducing the calculation time of simulation initialization and ensuring model convergence throughout transition processes.
摘要The increasing integration of cyber-physical components in Industry 4.0 water infrastructures has heightened the risk of false data injection(FDI)attacks,posing critical threats to operational integrity,resource management,and public safety.Traditional detection mechanisms often struggle to generalize across heterogeneous environments or adapt to sophisticated,stealthy threats.To address these challenges,we propose a novel evolutionary optimized transformer-based deep reinforcement learning framework(Evo-Transformer-DRL)designed for robust and adaptive FDI detection in smart water infrastructures.The proposed architecture integrates three powerful paradigms:a transformer encoder for modeling complex temporal dependencies in multivariate time series,a DRL agent for learning optimal decision policies in dynamic environments,and an evolutionary optimizer to fine-tune model hyper-parameters.This synergy enhances detection performance while maintaining adaptability across varying data distributions.Specifically,hyper-parameters of both the transformer and DRL modules are optimized using an improved grey wolf optimizer(IGWO),ensuring a balanced trade-off between detection accuracy and computational efficiency.The model is trained and evaluated on three realistic Industry 4.0 water datasets:secure water treatment(SWaT),water distribution(WADI),and battle of the attack detection algorithms(BATADAL),which capture diverse attack scenarios in smart treatment and distribution systems.Comparative analysis against state-of-the-art baselines including Transformer,DRL,bidirectional encoder representations from transformers(BERT),convolutional neural network(CNN),long short-term memory(LSTM),and support vector machines(SVM)demonstrates that our proposed Evo-Transformer-DRL framework consistently outperforms others in key metrics such as accuracy,recall,area under the curve(AUC),and execution time.Notably,it achieves a maximum detection accuracy of 99.19%,highlighting its strong generalization capability across different testbeds.These results confirm the suitability of our hybrid framework for real-world Industry 4.0 deployment,where rapid adaptation,scalability,and reliability are paramount for securing critical infrastructure systems.
基金supported by the National Key Research and Development Program of China(No.2022YFB4300902).
摘要As joint operations have become a key trend in modern military development,unmanned aerial vehicles(UAVs)play an increasingly important role in enhancing the intelligence and responsiveness of combat systems.However,the heterogeneity of aircraft,partial observability,and dynamic uncertainty in operational airspace pose significant challenges to autonomous collision avoidance using traditional methods.To address these issues,this paper proposes an adaptive collision avoidance approach for UAVs based on deep reinforcement learning.First,a unified uncertainty model incorporating dynamic wind fields is constructed to capture the complexity of joint operational environments.Then,to effectively handle the heterogeneity between manned and unmanned aircraft and the limitations of dynamic observations,a sector-based partial observation mechanism is designed.A Dynamic Threat Prioritization Assessment algorithm is also proposed to evaluate potential collision threats from multiple dimensions,including time to closest approach,minimum separation distance,and aircraft type.Furthermore,a Hierarchical Prioritized Experience Replay(HPER)mechanism is introduced,which classifies experience samples into high,medium,and low priority levels to preferentially sample critical experiences,thereby improving learning efficiency and accelerating policy convergence.Simulation results show that the proposed HPER-D3QN algorithm outperforms existing methods in terms of learning speed,environmental adaptability,and robustness,significantly enhancing collision avoidance performance and convergence rate.Finally,transfer experiments on a high-fidelity battlefield airspace simulation platform validate the proposed method's deployment potential and practical applicability in complex,real-world joint operational scenarios.
基金supported by the National Natural Science Foundation of China(Grant Nos.72571150 and 62306156)。
摘要Assessing the vulnerability of complex systems requires effective hypergraph dismantling strategies,yet existing methods struggle with the dynamic nature of cascading failures and the rugged optimization landscapes of high-order networks.In this paper,we propose a novel framework:hypergraph dismantling via evolutionary deep reinforcement learning(HD-EDR).First,we model a realistic dismantling environment incorporating hyperdegree-based and residual-capacitybased load redistribution mechanisms.Second,we introduce a hybrid learning architecture that synergizes the global exploration of evolutionary strategies with the gradient-based exploitation of deep reinforcement learning.A bidirectional parameter synchronization mechanism is designed to prevent the agent from being trapped in local optima.Furthermore,we integrate an inductive encoder to capture the evolving high-order dependencies of the residual network in real time.Extensive experiments across nine real-world datasets demonstrate that our framework significantly outperforms state-of-the-art baselines,providing a highly effective and robust strategy for maximizing structural damage in high-order networks.
基金supported in part by the Research on Key Technologies for the Development of an Active Balancing Cooperative Control Systemfor Distribution Networks and the National Natural Science Foundation of China under Grant 521532240029,Grant 62303006.
摘要To address the high costs and operational instability of distribution networks caused by the large-scale integration of distributed energy resources(DERs)(such as photovoltaic(PV)systems,wind turbines(WT),and energy storage(ES)devices),and the increased grid load fluctuations and safety risks due to uncoordinated electric vehicles(EVs)charging,this paper proposes a novel dual-scale hierarchical collaborative optimization strategy.This strategy decouples system-level economic dispatch from distributed EV agent control,effectively solving the resource coordination conflicts arising from the high computational complexity,poor scalability of existing centralized optimization,or the reliance on local information decision-making in fully decentralized frameworks.At the lower level,an EV charging and discharging model with a hybrid discrete-continuous action space is established,and optimized using an improved Parameterized Deep Q-Network(PDQN)algorithm,which directly handles mode selection and power regulation while embedding physical constraints to ensure safety.At the upper level,microgrid(MG)operators adopt a dynamic pricing strategy optimized through Deep Reinforcement Learning(DRL)to maximize economic benefits and achieve peak-valley shaving.Simulation results show that the proposed strategy outperforms traditional methods,reducing the total operating cost of the MG by 21.6%,decreasing the peak-to-valley load difference by 33.7%,reducing the number of voltage limit violations by 88.9%,and lowering the average electricity cost for EV users by 15.2%.This method brings a win-win result for operators and users,providing a reliable and efficient scheduling solution for distribution networks with high renewable energy penetration rates.
基金supported by the National Key Research and Development Program of China under grant 2021YFA0716600。
摘要In response to the rising demand for low-latency,computation-intensive applications in vehicular networks,this paper proposes an adaptive task offloading approach for Vehicle-to-Everything(V2X)environments.Leveraging an enhanced Multi-Agent Deep Deterministic Policy Gradient(MADDPG)algorithm with an attention mechanism,the proposed approach optimizes computation offloading and resource allocation,aiming to minimize energy consumption and service delay.In this paper,vehicles dynamically offload computing-intensive tasks to both nearby vehicles through V2V links and roadside units through V2I links.The adaptive attention mechanism enables the system to prioritize relevant state information,leading to faster convergence.Simulations conducted in a realistic urban V2X scenario demonstrate that the proposed Attention-enhanced MADDPG(AT-MADDPG)algorithm significantly improves performance,achieving notable reductions in both energy consumption and latency compared to baseline algorithms,especially in high-demand,dynamic scenarios.
基金supported by the Guizhou Provincial Key Technology R&D Program under Grant(QKHZC(2022)YB074)Guizhou University Science and Technology Group[2024]07.
摘要Mobile Edge Computing(MEC)facilitates the rapid response and energy-efficient execution of tasks on mobile devices.However,determining whether and where to offload tasks remains a significant challenge due to the constantly changing character of workloads in MEC environments.To address this issue,this paper proposes PreAlloc-A2C—a deep reinforcement learning actor-critic-based framework that calculates allocation scores by leveraging both task features(task size,required completion time,and waiting time)and server features(queue length and historical workload).This design enables fully distributed task offloading decisions without centralized coordination.Additionally,a Long Short-Term Memory(LSTM)network is integrated to forecast impending server loads,thereby supporting adaptive scheduling.A tailored reward function is also designed to jointly optimize three key performance metrics:task delay,device energy consumption,and task drop rate.Extensive experiments are conducted to evaluate PreAlloc-A2C against five baseline algorithms:Particle Swarm Optimization(PSO),Advantage Actor-Critic(A2C),Deep Q-Network(DQN),Double Deep Q-Network(DDQN),and Dueling Deep Q-Network(Dueling DQN).The results show that PreAlloc-A2C outperforms all baselines,achieving lower latency,reduced energy consumption,and a lower task drop rate.
基金supported by theHubei Provincial Technology Innovation Special Project and the Natural Science Foundation of Hubei Province under Grants 2023BEB024,2024AFC066,respectively.
摘要Underwater images frequently suffer from chromatic distortion,blurred details,and low contrast,posing significant challenges for enhancement.This paper introduces AquaTree,a novel underwater image enhancement(UIE)method that reformulates the task as a Markov Decision Process(MDP)through the integration of Monte Carlo Tree Search(MCTS)and deep reinforcement learning(DRL).The framework employs an action space of 25 enhancement operators,strategically grouped for basic attribute adjustment,color component balance,correction,and deblurring.Exploration within MCTS is guided by a dual-branch convolutional network,enabling intelligent sequential operator selection.Our core contributions include:(1)a multimodal state representation combining CIELab color histograms with deep perceptual features,(2)a dual-objective reward mechanism optimizing chromatic fidelity and perceptual consistency,and(3)an alternating training strategy co-optimizing enhancement sequences and network parameters.We further propose two inference schemes:an MCTS-based approach prioritizing accuracy at higher computational cost,and an efficient network policy enabling real-time processing with minimal quality loss.Comprehensive evaluations on the UIEB Dataset and Color correction and haze removal comparisons on the U45 Dataset demonstrate AquaTree’s superiority,significantly outperforming nine state-of-the-art methods across five established underwater image quality metrics.
摘要As the types of traffic requests increase,the elastic optical network(EON)is considered as a promising architecture to carry multiple types of traffic requests simultaneously,including immediate reservation(IR)and advance reservation(AR).Various resource allocation schemes for IR/AR requests have been designed in EON to reduce bandwidth blocking probability(BBP).However,these schemes do not consider different transmission requirements of IR requests and cannot maintain a low BBP for high-priority requests.In this paper,multi-priority is considered in the hybrid IR/AR request scenario.We modify the asynchronous advantage actor critic(A3C)model and propose an A3C-assisted priority resource allocation(APRA)algorithm.The APRA integrates priority and transmission quality of IR requests to design the A3C reward function,then dynamically allocates dedicated resources for different IR requests according to the time-varying requirements.By maximizing the reward,the transmission quality of IR requests can be matched with the priority,and lower BBP for high-priority IR requests can be ensured.Simulation results show that the APRA reduces the BBP of high-priority IR requests from 0.0341 to0.0138,and the overall network operation gain is improved by 883 compared to the scheme without considering the priority.
摘要At present,energy consumption is one of the main bottlenecks in autonomous mobile robot development.To address the challenge of high energy consumption in path planning for autonomous mobile robots navigating unknown and complex environments,this paper proposes an Attention-Enhanced Dueling Deep Q-Network(ADDueling DQN),which integrates a multi-head attention mechanism and a prioritized experience replay strategy into a Dueling-DQN reinforcement learning framework.A multi-objective reward function,centered on energy efficiency,is designed to comprehensively consider path length,terrain slope,motion smoothness,and obstacle avoidance,enabling optimal low-energy trajectory generation in 3D space from the source.The incorporation of a multihead attention mechanism allows the model to dynamically focus on energy-critical state features—such as slope gradients and obstacle density—thereby significantly improving its ability to recognize and avoid energy-intensive paths.Additionally,the prioritized experience replay mechanism accelerates learning from key decision-making experiences,suppressing inefficient exploration and guiding the policy toward low-energy solutions more rapidly.The effectiveness of the proposed path planning algorithm is validated through simulation experiments conducted in multiple off-road scenarios.Results demonstrate that AD-Dueling DQN consistently achieves the lowest average energy consumption across all tested environments.Moreover,the proposed method exhibits faster convergence and greater training stability compared to baseline algorithms,highlighting its global optimization capability under energy-aware objectives in complex terrains.This study offers an efficient and scalable intelligent control strategy for the development of energy-conscious autonomous navigation systems.
基金funded by the National Natural Science Foundation of China,grant number 62362052.
摘要This paper investigates the joint resource allocation problem in Reconfigurable Intelligent Surface(RIS)-assisted cooperative non-orthogonal multiple access device-to-device(CNOMA-D2D)cellular networks.To tackle the high-dimensional non-convex joint optimization of power control,RIS phase configuration and channel assignment,we propose an integrated user pairing strategy,PIP-UP,quantifying utility through factors,phase alignment,interference suppression and power difference,neglected in existing methods.Furthermore,we develop a hybrid deep reinforcement learning algorithm,A3TD,combining the parallel exploration capability of Asynchronous Advantage Actor-Critic(A3C)with the stable continuous optimization of Twin Delayed Deep Deterministic Policy Gradient(TD3).This integration enables efficient and robust joint optimization of D2D channel allocation,transmit power,and RIS phase shifts.Simulation results demonstrate that the proposed A3TD algorithm significantly outperforms baseline algorithms,Actor-Critic(AC),Deep Deterministic Policy Gradient(DDPG)and TD3,in terms of sum rate and convergence speed,validating its effectiveness for resource management in complex RIS-assisted CNOMA-D2D networks.
基金funded by the National Defense Science and Technology Innovation project,grant number ZZKY20253129.
摘要Deep reinforcement learning(DRL)has become an important method in Unmanned Aerial Vehicle(UAV)path planning,but the field still lacks a dedicated bibliometric review that summarizes its publication patterns,intellectual structure,and thematic evolution.This study analyzes 1402 Web of Science publications from 2010 to 2025 using CiteSpace,VOSviewer,and the Bibliometrix R package.Three main findings are reported.First,the bibliometric evidence suggests a four-phase evolution of the field—foundational exploration(2015–2016),continuous-control breakthrough(2017–2019),multi-agent collaborative coordination(2020–2022),and complexscenario integration(2023–2025)—as reflected in publication trends,keyword bursts,and co-citation clusters.Second,co-citation and keyword analyses indicate a gradual shift from geometric navigation toward the joint consideration of communication,energy,and mission objectives,a pattern also reflected in the prominence of Internet of Things(IoT)and vehicular-technology journals.Third,burst and clustering results highlight several active research directions,including multi-objective cooperative decision-making,hierarchical planning architectures that combine global perception with local control,and communication–energy co-design.Rather than offering prescriptive conclusions,these results provide descriptive bibliometric evidence that may help researchers understand the development and emerging priorities of DRL-based UAV path planning.