UM CONTROLADOR ADAPTATIVO ÓTIMO BASEADO EM APRENDIZADO ONLINE ATOR-CRÍTICO PARA UM MANIPULADOR ROBÓTICO
DOI:
https://doi.org/10.56238/bocav25n74-031Palavras-chave:
Manipulador Robótico, Controle Adaptativo, Controle Ótimo, Aprendizado por Reforço, Esquema Ator-CríticoResumo
As incertezas nos parâmetros de um manipulador robótico podem afetar, de forma significativa, o desempenho do manipulador, ocasionando erros de regime e de seguimento de trajetória. Controladores adaptativos apresentam-se como uma boa alternativa para esses sistemas, pois possuem como principal característica a capacidade de aprenderem online usando estimação de parâmetros em tempo real. No entanto, controladores adaptativos não são geralmente projetados com a qualidade de serem ótimos com respeito aos critérios de desempenho especificados e, desta forma, não são viáveis para aplicações onde o uso ótimo de recursos é altamente desejável, como por exemplo em robôs humanoides e robôs de serviços. Este artigo apresenta o projeto e investigação de desempenho de um controlador que combina características de controle adaptativo e controle ótimo para um manipulador robótico. Especificamente, o esquema de controle proposto é implementado como uma estrutura ator-crítico, a qual está inserida no contexto de aprendizado por reforço, caracterizando este projeto como uma abordagem independente do modelo da planta. Em contraste a outros sistemas ator-críticos em que são usadas duas redes neurais independentes, uma para aproximar a função valor, e a outra para aprender ações de controle, neste esquema, se define uma única rede neural, o que reduz o número de parâmetros a serem estimados. Os resultados de simulação demonstram o desempenho desejado do controlador proposto que atua em um manipulador de juntas rotativas com dois graus de liberdade.
Referências
ABBAS, Z. Motion control of robotic arm manipulator using PID and sliding mode technique. 2018. Tese (Doutorado em Engenharia Elétrica) – Capital University of Science and Technology, Islamabad, 2018.
AL-OLIMAT, K. S.; GHANDAKLY, A. A. Multiple model reference adaptive control algorithm using on-line fuzzy logic adjustment and its application to robotic manipulators. In: Conference Record of the 2002 IEEE Industry Applications Conference. 37th IAS Annual Meeting, Pittsburgh, PA, USA, 2002, p. 1463-1466. DOI: https://doi.org/10.1109/IAS.2002.1042748
ALQAUDI, B. et al. Model reference adaptive impedance control for physical human-robot interaction. Control Theory and Technology. v. 14, p. 68-82, fev. 2016. DOI: https://doi.org/10.1007/s11768-016-5138-2
BHATNAGAR, S. et al. Natural actor-critic algorithms. Automatica, v. 45, n. 11, p. 2471-2482, nov. 2009. DOI: https://doi.org/10.1016/j.automatica.2009.07.008
BORASE, R. P. et al. A review of PID control, tuning methods and applications. International Journal of Dynamics and Control. v. 9, p. 818-827, 2021. DOI: https://doi.org/10.1007/s40435-020-00665-4
CAO, S. et al. Reinforcement learning-based fixed-time trajectory tracking control for uncertain robotic manipulators with input saturation. IEEE Transactions on Neural Networks and Learning Systems, v. 34, n. 8, p. 4584-4595, ago. 2023. DOI: https://doi.org/10.1109/TNNLS.2021.3116713
CHEN, L.; DAI, S.-L.; DONG, C. Adaptive optimal tracking control of an underactuated surface vessel using actor–critic reinforcement learning. IEEE Transactions on Neural Networks and Learning Systems, v. 35, n. 6, p. 7520-7533, jun. 2024a. DOI: https://doi.org/10.1109/TNNLS.2022.3214681
CHEN, L.; DONG, C.; DAI, S.-L. Adaptive optimal consensus control of multiagent systems with unknown dynamics and disturbances via reinforcement learning. IEEE Transactions on Artificial Intelligence, v. 5, n. 5, p. 2193-2203, maio 2024b. DOI: https://doi.org/10.1109/TAI.2023.3324895
CHEN, W.-D. Experimental study of robot manipulators based on robust adaptive control. In: International Conference on Machine Learning and Cybernetics, Guangzhou, China, 2005, p. 18-21.
CLEGG, A. C.; DUNNIGAN, M. W.; LANE, D. M. Self-tuning position and force control of an underwater hydraulic manipulator. In: Proceedings 2001 ICRA. IEEE International Conference on Robotics and Automation, Seoul, Korea (South), maio. 2001, p. 3226-3231. DOI: https://doi.org/10.1109/ROBOT.2001.933115
CRAIG, J. J. Introduction to robotics: mechanics and control, Ed. 4. Global Edition. São Paulo: Pearson, 2021.
DUBOWSKY, S.; DESFORGES, D. T. The application of model-referenced adaptive control to robotic manipulators. Journal of Dynamic Systems, Measurement, and Control, v. 101, n. 3, p. 193-200, set. 1979. DOI: https://doi.org/10.1115/1.3426424
FATEH, S.; FATEH, M. M. Adaptive fuzzy control of robot manipulators with asymptotic tracking performance. Journal of Control, Automation and Electrical Systems, v. 31, p. 52-61, out. 2019. DOI: https://doi.org/10.1007/s40313-019-00496-5
FERREIRA, E. F. M.; RÊGO, P. H. M.; NETO, J. V. F. Numerical stability improvements of state-value function approximations based on RLS learning for online HDP-DLQR control system design. Engineering Applications of Artificial Intelligence, v. 63, p.1-19, ago. 2017. DOI: https://doi.org/10.1016/j.engappai.2017.04.017
FREIRE, E. O.; ROSSOMANDO, F. G; SORIA, C. M. Self-tuning of a neuro-adaptive PID controller for a SCARA robot based on neural network. IEEE Latin America Transactions, v. 16, n. 5, p. 1364-1374, jul. 2018. DOI: https://doi.org/10.1109/TLA.2018.8408429
GUO, X.; YAN, W.; CUI, R. Reinforcement learning-based nearly optimal control for constrained-input partially unknown systems using differentiator. IEEE Transactions on Neural Networks and Learning Systems, v. 31, n. 11, p. 4713-4725, nov. 2020. DOI: https://doi.org/10.1109/TNNLS.2019.2957287
HE, W. et al. Reinforcement learning control of a flexible two-link manipulator: an experimental investigation. IEEE Transactions on Systems, Man, and Cybernetics: Systems, v. 51, n. 12, p. 7326-7336, dez. 2021. DOI: https://doi.org/10.1109/TSMC.2020.2975232
HU, Q.; XU, L.; ZHANG, A. Adaptive backstepping trajectory tracking control of robot manipulator. Journal of the Franklin Institute. v. 349, n. 3, p. 1087-1105, 2012. DOI: https://doi.org/10.1016/j.jfranklin.2012.01.001
HU, Y.; SI, B. A reinforcement learning neural network for robotic manipulator control. Neural Computation. v. 30, n. 7, p. 1983-2004, jul. 2018. DOI: https://doi.org/10.1162/neco_a_01079
JIANG, Y.; JIANG, Z.-P. Robust adaptive dynamic programming. Hoboken, New Jersey: John Wiley & Sons, Inc., 2017. DOI: https://doi.org/10.1002/9781119132677
KAMBOJ, A. et al. L. Discrete-time Lyapunov based kinematic control of robot manipulator using actor-critic framework. In: 2020 International Joint Conference on Neural Networks (IJCNN), Glasgow, United Kingdom, 2020, p. 1-7. DOI: https://doi.org/10.1109/IJCNN48605.2020.9207522
KHAN, S. G. et al. Reinforcement learning based compliance control of a robotic walk assist device, Advanced Robotics, v.33, n. 24, p. 1281-1292, nov. 2019. DOI: https://doi.org/10.1080/01691864.2019.1690574
KHAN, S. G. et al. A Q-learning based Cartesian model reference compliance controller implementation for a humanoid robot arm. In: 2011 IEEE 5th International Conference on Robotics, Automation and Mechatronics (RAM), Qingdao, China, set. 2011, p. 214-219. DOI: https://doi.org/10.1109/RAMECH.2011.6070484
KHAN, S. G. et al. Reinforcement learning and optimal adaptive control: an overview and implementation examples. Annual Reviews in Control. v. 36, n.1, p. 42-59, 2012. DOI: https://doi.org/10.1016/j.arcontrol.2012.03.004
KIUMARSI, B. et al. Optimal and autonomous control using reinforcement learning: a survey. IEEE Transactions on Neural Networks and Learning Systems, v. 29, n. 6, p. 2042-2062, jun. 2018. DOI: https://doi.org/10.1109/TNNLS.2017.2773458
KONSTANTOPOULOS, G. C.; BALDIVIESO-MONASTERIOS, P. R. State-limiting PID controller for a class of nonlinear systems with constant uncertainties. International Journal of Robust and Nonlinear Control, v. 30, p. 1770-1787, 2020. DOI: https://doi.org/10.1002/rnc.4853
MALIOTIS, G. A. Hybrid model reference adaptive control/computed torque control scheme for robotic manipulators. Proceedings of the Institution of Mechanical Engineers, Part I: Journal of Systems and Control Engineering, v. 205, n. 3, p. 215-21, 1991. DOI: https://doi.org/10.1243/PIME_PROC_1991_205_334_02
MOOSAVI, S. K. R.; ZAFAR, M. H.; SANFILIPPO, F. Forward kinematic modelling with radial basis function neural network tuned with a novel meta-heuristic algorithm for robotic manipulators. Robotics, v. 11, n. 2, p. 1-17, abr. 2022. DOI: https://doi.org/10.3390/robotics11020043
PANE, Y. P. et al. Reinforcement learning based compensation methods for robot manipulators. Engineering Applications of Artificial Intelligence, v. 78, p. 236-247, fev. 2019. DOI: https://doi.org/10.1016/j.engappai.2018.11.006
PANE, Y. P.; NAGESHRAO, S. P.; BABUŠKA, R. Actor-critic reinforcement learning for tracking control in robotics. In: 2016 IEEE 55th Conference on Decision and Control (CDC), Las Vegas, NV, USA, dez. 2016, p. 5819-5826. DOI: https://doi.org/10.1109/CDC.2016.7799164
PETERS, J.; SCHAAL, S. Learning to control in operational space. International Journal of Robotics Research, v. 27, n. 2, p. 197-212, fev. 2008a. DOI: https://doi.org/10.1177/0278364907087548
PETERS, J.; SCHAAL, S. Natural actor-critic. Neurocomputing, v. 71, n. 7-9, p. 1180-1190, marc. 2008b. DOI: https://doi.org/10.1016/j.neucom.2007.11.026
PLUŠKOSKI, A.; CIGANOVIĆ, I.; JOVANOVIĆ, M. D. Benefits of Residual Networks in Reinforcement Learning using V-Rep Simulator. In: 6th International Conference IcETRAN, Silver Lake, Serbia, jun. 2019, p. 1-6.
QI, R.; TAO, G.; JIANG, B. Adaptive control: a tutorial introduction. In book: Fuzzy system identification and adaptive control. Communications and Control Engineering. Springer, Cham., 2019, p. 55-74. DOI: https://doi.org/10.1007/978-3-030-19882-4_3
QUIGLEY, M.; GERKEY, B.; SMART, W. D. Programming robots with ROS: a practical introduction to the robot operating system. Ed. 1. O'Reilly Media, Inc. 2015.
ROHMER, E.; SINGH, S. P. N.; FREESE, M. V-REP: A versatile and scalable robot simulation framework. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, Tokyo, Japan, nov. 2013. DOI: https://doi.org/10.1109/IROS.2013.6696520
SASAKI, M. et al. Self-tuning control of a two-link flexible manipulator using neural networks. In: 2009 ICCAS-SICE, Fukuoka, Japan, ago. 2009, p. 2468-2473.
SHAH, H.; GOPAL, M. Reinforcement learning control of robot manipulators in uncertain environments. In: IEEE International Conference on Industrial Technology, Churchill, VIC, Australia, fev. 2009, p. 1-6. DOI: https://doi.org/10.1109/ICIT.2009.4939504
SHAMSHIRI, R. R. et al. Robotic harvesting of fruiting vegetables: A simulation approach in V-REP, ROS and MATLAB. In (Ed.), Automation in Agriculture - Securing Food Supplies for Future Generations. IntechOpen. mar. 2018. DOI: https://doi.org/10.5772/intechopen.73861
SU, Y. et al. Fixed-time optimal trajectory tracking control for an unmanned surface vehicle via reinforcement learning. IEEE/ASME Transactions on Mechatronics, p. 1-12, set. 2025. DOI: https://doi.org/10.1109/TMECH.2025.3602024
SUN, N. et al. Adaptive control for pneumatic artificial muscle systems with parametric uncertainties and unidirectional input constraints. IEEE Transactions on Industrial Informatics, v. 16, n. 2, p. 969-979, fev. 2020. DOI: https://doi.org/10.1109/TII.2019.2923715
SUTTON, R. S.; BARTO, A. G. Reinforcement learning: An Introduction. Ed. 2. Cambridge, Massachusetts: MIT Press, 2018.
VRABIE, D.; VAMVOUDAKIS, K. G.; LEWIS, F. L. Optimal adaptive control and differential games by reinforcement learning principles. London, United Kingdom: The Institution of Engineering and Technology, 2013. DOI: https://doi.org/10.1049/PBCE081E
WANG, Z. et al. Adaptive altitude control for underwater vehicles based on deep reinforcement learning. In: 2025 8th International Conference on Transportation Information and Safety (ICTIS), Granada, Spain, 2025, p. 79-84. DOI: https://doi.org/10.1109/ICTIS68762.2025.11214834
WU, L.; YAN, Q; CAI, J. Neural network-based adaptive learning control for robot manipulators with arbitrary initial errors. IEEE Access, v. 7, p. 180194-180204, dez. 2019. DOI: https://doi.org/10.1109/ACCESS.2019.2958371
YAGHMAIE, F. A.; GUSTAFSSON, F.; LJUNG , L. Linear quadratic control using model-free reinforcement learning. IEEE Transactions on Automatic Control, v. 68, n. 2, p. 737-752, fev. 2023. DOI: https://doi.org/10.1109/TAC.2022.3145632
YILMAZ, B. M. et al. Self-adjusting fuzzy logic based control of robot manipulators in task space. IEEE Transactions on Industrial Electronics, v. 69, n. 2, p. 1620-1629, fev. 2022. DOI: https://doi.org/10.1109/TIE.2021.3063970
ZHANG, D.; WEI, B. Design, analysis and modelling of a hybrid controller for serial robotic manipulators. Robotica, v. 35, n. 9, p. 1888-1905, set. 2017. DOI: https://doi.org/10.1017/S0263574716000564
ZHAO, D. et al. Linear quadratic control of unknown nonlinear systems using model-free reinforcement learning. IEEE Transactions on Industrial Electronics, v. 72, n. 12, p. 13751-13762, dez. 2025. DOI: https://doi.org/10.1109/TIE.2025.3581264
Downloads
Publicado
Edição
Seção
Licença
Este trabalho está licenciado sob uma licença Creative Commons Attribution 4.0 International License.
Direitos autorais (c) .
Esta obra está licenciada sob uma licença Creative Commons Atribuição 4.0 Internacional.
