AN OPTIMAL ADAPTIVE CONTROLLER BASED ON ONLINE ACTOR-CRITIC LEARNING FOR A ROBOT MANIPULATOR
DOI:
https://doi.org/10.56238/bocav25n74-031Keywords:
Robot Manipulator, Adaptive Control, Optimal Control, Reinforcement Learning, Actor-Critic SchemeAbstract
The uncertainties in the parameters of a robot manipulator can significantly affect the robot performance, causing steady-state and trajectory following errors. Adaptive controllers are a good alternative for these systems, since their main feature is the capability to learn online using real-time parameter estimation. Nevertheless, adaptive controllers are not usually designed to be optimal to a prescribed performance index, and thus, are not suitable to applications in which optimal use of resources is highly desirable, for instance humanoid and service robots. This paper presents the design and performance study of a controller that combine features of adaptive control and optimal control applied to a robot manipulator. Specifically, the proposed control scheme is implemented as an actor-critic structure, which is in the reinforcement learning context, characterizing this design as a model-free approach. In contrast to others actor-critic systems in which two independent neural networks are used, one for approximating the value function and another for learning the control actions, in this scheme, a single neural network is defined, reducing the number of parameters to be estimated. The simulation results validate the desired performance of the proposed controller applied in a two-link robot manipulator with revolute joints.
References
ABBAS, Z. Motion control of robotic arm manipulator using PID and sliding mode technique. 2018. Tese (Doutorado em Engenharia Elétrica) – Capital University of Science and Technology, Islamabad, 2018.
AL-OLIMAT, K. S.; GHANDAKLY, A. A. Multiple model reference adaptive control algorithm using on-line fuzzy logic adjustment and its application to robotic manipulators. In: Conference Record of the 2002 IEEE Industry Applications Conference. 37th IAS Annual Meeting, Pittsburgh, PA, USA, 2002, p. 1463-1466. DOI: https://doi.org/10.1109/IAS.2002.1042748
ALQAUDI, B. et al. Model reference adaptive impedance control for physical human-robot interaction. Control Theory and Technology. v. 14, p. 68-82, fev. 2016. DOI: https://doi.org/10.1007/s11768-016-5138-2
BHATNAGAR, S. et al. Natural actor-critic algorithms. Automatica, v. 45, n. 11, p. 2471-2482, nov. 2009. DOI: https://doi.org/10.1016/j.automatica.2009.07.008
BORASE, R. P. et al. A review of PID control, tuning methods and applications. International Journal of Dynamics and Control. v. 9, p. 818-827, 2021. DOI: https://doi.org/10.1007/s40435-020-00665-4
CAO, S. et al. Reinforcement learning-based fixed-time trajectory tracking control for uncertain robotic manipulators with input saturation. IEEE Transactions on Neural Networks and Learning Systems, v. 34, n. 8, p. 4584-4595, ago. 2023. DOI: https://doi.org/10.1109/TNNLS.2021.3116713
CHEN, L.; DAI, S.-L.; DONG, C. Adaptive optimal tracking control of an underactuated surface vessel using actor–critic reinforcement learning. IEEE Transactions on Neural Networks and Learning Systems, v. 35, n. 6, p. 7520-7533, jun. 2024a. DOI: https://doi.org/10.1109/TNNLS.2022.3214681
CHEN, L.; DONG, C.; DAI, S.-L. Adaptive optimal consensus control of multiagent systems with unknown dynamics and disturbances via reinforcement learning. IEEE Transactions on Artificial Intelligence, v. 5, n. 5, p. 2193-2203, maio 2024b. DOI: https://doi.org/10.1109/TAI.2023.3324895
CHEN, W.-D. Experimental study of robot manipulators based on robust adaptive control. In: International Conference on Machine Learning and Cybernetics, Guangzhou, China, 2005, p. 18-21.
CLEGG, A. C.; DUNNIGAN, M. W.; LANE, D. M. Self-tuning position and force control of an underwater hydraulic manipulator. In: Proceedings 2001 ICRA. IEEE International Conference on Robotics and Automation, Seoul, Korea (South), maio. 2001, p. 3226-3231. DOI: https://doi.org/10.1109/ROBOT.2001.933115
CRAIG, J. J. Introduction to robotics: mechanics and control, Ed. 4. Global Edition. São Paulo: Pearson, 2021.
DUBOWSKY, S.; DESFORGES, D. T. The application of model-referenced adaptive control to robotic manipulators. Journal of Dynamic Systems, Measurement, and Control, v. 101, n. 3, p. 193-200, set. 1979. DOI: https://doi.org/10.1115/1.3426424
FATEH, S.; FATEH, M. M. Adaptive fuzzy control of robot manipulators with asymptotic tracking performance. Journal of Control, Automation and Electrical Systems, v. 31, p. 52-61, out. 2019. DOI: https://doi.org/10.1007/s40313-019-00496-5
FERREIRA, E. F. M.; RÊGO, P. H. M.; NETO, J. V. F. Numerical stability improvements of state-value function approximations based on RLS learning for online HDP-DLQR control system design. Engineering Applications of Artificial Intelligence, v. 63, p.1-19, ago. 2017. DOI: https://doi.org/10.1016/j.engappai.2017.04.017
FREIRE, E. O.; ROSSOMANDO, F. G; SORIA, C. M. Self-tuning of a neuro-adaptive PID controller for a SCARA robot based on neural network. IEEE Latin America Transactions, v. 16, n. 5, p. 1364-1374, jul. 2018. DOI: https://doi.org/10.1109/TLA.2018.8408429
GUO, X.; YAN, W.; CUI, R. Reinforcement learning-based nearly optimal control for constrained-input partially unknown systems using differentiator. IEEE Transactions on Neural Networks and Learning Systems, v. 31, n. 11, p. 4713-4725, nov. 2020. DOI: https://doi.org/10.1109/TNNLS.2019.2957287
HE, W. et al. Reinforcement learning control of a flexible two-link manipulator: an experimental investigation. IEEE Transactions on Systems, Man, and Cybernetics: Systems, v. 51, n. 12, p. 7326-7336, dez. 2021. DOI: https://doi.org/10.1109/TSMC.2020.2975232
HU, Q.; XU, L.; ZHANG, A. Adaptive backstepping trajectory tracking control of robot manipulator. Journal of the Franklin Institute. v. 349, n. 3, p. 1087-1105, 2012. DOI: https://doi.org/10.1016/j.jfranklin.2012.01.001
HU, Y.; SI, B. A reinforcement learning neural network for robotic manipulator control. Neural Computation. v. 30, n. 7, p. 1983-2004, jul. 2018. DOI: https://doi.org/10.1162/neco_a_01079
JIANG, Y.; JIANG, Z.-P. Robust adaptive dynamic programming. Hoboken, New Jersey: John Wiley & Sons, Inc., 2017. DOI: https://doi.org/10.1002/9781119132677
KAMBOJ, A. et al. L. Discrete-time Lyapunov based kinematic control of robot manipulator using actor-critic framework. In: 2020 International Joint Conference on Neural Networks (IJCNN), Glasgow, United Kingdom, 2020, p. 1-7. DOI: https://doi.org/10.1109/IJCNN48605.2020.9207522
KHAN, S. G. et al. Reinforcement learning based compliance control of a robotic walk assist device, Advanced Robotics, v.33, n. 24, p. 1281-1292, nov. 2019. DOI: https://doi.org/10.1080/01691864.2019.1690574
KHAN, S. G. et al. A Q-learning based Cartesian model reference compliance controller implementation for a humanoid robot arm. In: 2011 IEEE 5th International Conference on Robotics, Automation and Mechatronics (RAM), Qingdao, China, set. 2011, p. 214-219. DOI: https://doi.org/10.1109/RAMECH.2011.6070484
KHAN, S. G. et al. Reinforcement learning and optimal adaptive control: an overview and implementation examples. Annual Reviews in Control. v. 36, n.1, p. 42-59, 2012. DOI: https://doi.org/10.1016/j.arcontrol.2012.03.004
KIUMARSI, B. et al. Optimal and autonomous control using reinforcement learning: a survey. IEEE Transactions on Neural Networks and Learning Systems, v. 29, n. 6, p. 2042-2062, jun. 2018. DOI: https://doi.org/10.1109/TNNLS.2017.2773458
KONSTANTOPOULOS, G. C.; BALDIVIESO-MONASTERIOS, P. R. State-limiting PID controller for a class of nonlinear systems with constant uncertainties. International Journal of Robust and Nonlinear Control, v. 30, p. 1770-1787, 2020. DOI: https://doi.org/10.1002/rnc.4853
MALIOTIS, G. A. Hybrid model reference adaptive control/computed torque control scheme for robotic manipulators. Proceedings of the Institution of Mechanical Engineers, Part I: Journal of Systems and Control Engineering, v. 205, n. 3, p. 215-21, 1991. DOI: https://doi.org/10.1243/PIME_PROC_1991_205_334_02
MOOSAVI, S. K. R.; ZAFAR, M. H.; SANFILIPPO, F. Forward kinematic modelling with radial basis function neural network tuned with a novel meta-heuristic algorithm for robotic manipulators. Robotics, v. 11, n. 2, p. 1-17, abr. 2022. DOI: https://doi.org/10.3390/robotics11020043
PANE, Y. P. et al. Reinforcement learning based compensation methods for robot manipulators. Engineering Applications of Artificial Intelligence, v. 78, p. 236-247, fev. 2019. DOI: https://doi.org/10.1016/j.engappai.2018.11.006
PANE, Y. P.; NAGESHRAO, S. P.; BABUŠKA, R. Actor-critic reinforcement learning for tracking control in robotics. In: 2016 IEEE 55th Conference on Decision and Control (CDC), Las Vegas, NV, USA, dez. 2016, p. 5819-5826. DOI: https://doi.org/10.1109/CDC.2016.7799164
PETERS, J.; SCHAAL, S. Learning to control in operational space. International Journal of Robotics Research, v. 27, n. 2, p. 197-212, fev. 2008a. DOI: https://doi.org/10.1177/0278364907087548
PETERS, J.; SCHAAL, S. Natural actor-critic. Neurocomputing, v. 71, n. 7-9, p. 1180-1190, marc. 2008b. DOI: https://doi.org/10.1016/j.neucom.2007.11.026
PLUŠKOSKI, A.; CIGANOVIĆ, I.; JOVANOVIĆ, M. D. Benefits of Residual Networks in Reinforcement Learning using V-Rep Simulator. In: 6th International Conference IcETRAN, Silver Lake, Serbia, jun. 2019, p. 1-6.
QI, R.; TAO, G.; JIANG, B. Adaptive control: a tutorial introduction. In book: Fuzzy system identification and adaptive control. Communications and Control Engineering. Springer, Cham., 2019, p. 55-74. DOI: https://doi.org/10.1007/978-3-030-19882-4_3
QUIGLEY, M.; GERKEY, B.; SMART, W. D. Programming robots with ROS: a practical introduction to the robot operating system. Ed. 1. O'Reilly Media, Inc. 2015.
ROHMER, E.; SINGH, S. P. N.; FREESE, M. V-REP: A versatile and scalable robot simulation framework. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, Tokyo, Japan, nov. 2013. DOI: https://doi.org/10.1109/IROS.2013.6696520
SASAKI, M. et al. Self-tuning control of a two-link flexible manipulator using neural networks. In: 2009 ICCAS-SICE, Fukuoka, Japan, ago. 2009, p. 2468-2473.
SHAH, H.; GOPAL, M. Reinforcement learning control of robot manipulators in uncertain environments. In: IEEE International Conference on Industrial Technology, Churchill, VIC, Australia, fev. 2009, p. 1-6. DOI: https://doi.org/10.1109/ICIT.2009.4939504
SHAMSHIRI, R. R. et al. Robotic harvesting of fruiting vegetables: A simulation approach in V-REP, ROS and MATLAB. In (Ed.), Automation in Agriculture - Securing Food Supplies for Future Generations. IntechOpen. mar. 2018. DOI: https://doi.org/10.5772/intechopen.73861
SU, Y. et al. Fixed-time optimal trajectory tracking control for an unmanned surface vehicle via reinforcement learning. IEEE/ASME Transactions on Mechatronics, p. 1-12, set. 2025. DOI: https://doi.org/10.1109/TMECH.2025.3602024
SUN, N. et al. Adaptive control for pneumatic artificial muscle systems with parametric uncertainties and unidirectional input constraints. IEEE Transactions on Industrial Informatics, v. 16, n. 2, p. 969-979, fev. 2020. DOI: https://doi.org/10.1109/TII.2019.2923715
SUTTON, R. S.; BARTO, A. G. Reinforcement learning: An Introduction. Ed. 2. Cambridge, Massachusetts: MIT Press, 2018.
VRABIE, D.; VAMVOUDAKIS, K. G.; LEWIS, F. L. Optimal adaptive control and differential games by reinforcement learning principles. London, United Kingdom: The Institution of Engineering and Technology, 2013. DOI: https://doi.org/10.1049/PBCE081E
WANG, Z. et al. Adaptive altitude control for underwater vehicles based on deep reinforcement learning. In: 2025 8th International Conference on Transportation Information and Safety (ICTIS), Granada, Spain, 2025, p. 79-84. DOI: https://doi.org/10.1109/ICTIS68762.2025.11214834
WU, L.; YAN, Q; CAI, J. Neural network-based adaptive learning control for robot manipulators with arbitrary initial errors. IEEE Access, v. 7, p. 180194-180204, dez. 2019. DOI: https://doi.org/10.1109/ACCESS.2019.2958371
YAGHMAIE, F. A.; GUSTAFSSON, F.; LJUNG , L. Linear quadratic control using model-free reinforcement learning. IEEE Transactions on Automatic Control, v. 68, n. 2, p. 737-752, fev. 2023. DOI: https://doi.org/10.1109/TAC.2022.3145632
YILMAZ, B. M. et al. Self-adjusting fuzzy logic based control of robot manipulators in task space. IEEE Transactions on Industrial Electronics, v. 69, n. 2, p. 1620-1629, fev. 2022. DOI: https://doi.org/10.1109/TIE.2021.3063970
ZHANG, D.; WEI, B. Design, analysis and modelling of a hybrid controller for serial robotic manipulators. Robotica, v. 35, n. 9, p. 1888-1905, set. 2017. DOI: https://doi.org/10.1017/S0263574716000564
ZHAO, D. et al. Linear quadratic control of unknown nonlinear systems using model-free reinforcement learning. IEEE Transactions on Industrial Electronics, v. 72, n. 12, p. 13751-13762, dez. 2025. DOI: https://doi.org/10.1109/TIE.2025.3581264
Downloads
Published
Issue
Section
License
This work is licensed under a Creative Commons Attribution 4.0 International License.
Copyright (c). Conjuncture Bulletin (BOCA)
This work is licensed under a Creative Commons Attribution 4.0 International License.
