Skip to main content
Springer Nature Link
Log in

Optimal time-varying formation tracking control for nonlinear multi-agent systems via event-triggered and self-triggered reinforcement learning

  • Research
  • Published:
Nonlinear Dynamics Aims and scope Submit manuscript

Abstract

This paper investigates the optimal time-varying formation (TVF) tracking control problem for discrete-time (DT) second-order nonlinear multi-agent systems (NMASs). A reinforcement learning (RL) algorithm is developed under both event-triggered (ET) and self-triggered (ST) mechanisms. Firstly, an event-triggering mechanism (ETM) is designed using Lyapunov function method, where the triggering threshold depends on the agent’s own triggering state and the input information from neighbor agents at event-triggering instants. An ET-based policy iteration (PI) algorithm is then presented to solve the ET-based discrete-time Hamilton-Jacobi-Bellman equation (HJB), thereby deriving the optimal ET control strategy. To facilitate online implementation, an actor-critic neural networks (AC NNs) framework is proposed to approximate the performance index function and learn the optimal strategy, with the actor network weights updated only at triggering instants. Theoretical analysis confirms that the formation tracking errors and weight estimation errors are uniformly ultimately bounded (UUB). Moreover, a ST approach is introduced to eliminate the need for continuous state monitoring at each DT step. Finally, the effectiveness of the proposed methods is illustrated through a simulation example.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+
from €37.37 /Month
  • Starting from 10 chapters or articles per month
  • Access and download chapters and articles from more than 300k books and 2,500 journals
  • Cancel anytime
View plans

Buy Now

Price includes VAT (Norway)

Instant access to the full article PDF.

Institutional subscriptions

Algorithm 1
Algorithm 2
Algorithm 3
Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Explore related subjects

Discover the latest articles, books and news in related subjects, suggested using machine learning.

Data Availability

No datasets were generated or analysed during the current study.

References

  1. Gong, P., Wang, Q.-G.: Robust adaptive distributed optimization for heterogeneous unknown second-order nonlinear multiagent systems. Sci. China Inf. Sci. 68(4), 149202 (2025)

    Article  MathSciNet  Google Scholar 

  2. Derakhshan, F., Yousefi, S.: A review on the applications of multiagent systems in wireless sensor networks. Int. J. Distrib. Sens. Netw. 15(5), 155014771985076 (2019)

    Article  Google Scholar 

  3. Zhu, W., Jiang, Z.-P.: Event-based leader-following consensus of multi-agent systems with input time delay. IEEE Trans. Autom. Control 60(5), 1362–1367 (2015)

    Article  ADS  MathSciNet  Google Scholar 

  4. Dong, X., Hu, G.: Time-varying formation tracking for linear multiagent systems with multiple leaders. IEEE Trans. Autom. Control 62(7), 3658–3664 (2017)

    Article  ADS  MathSciNet  Google Scholar 

  5. Fu, L., Zheng, W., Wang, H.: Nonlinear heterogeneous multi-agent systems under communication switching environment bipartite formation tracking control. Nonlinear Dyn. 113(10), 11817–11828 (2025)

    Article  Google Scholar 

  6. Wen, G., Chen, C.L.P., Li, B.: Optimized formation control using simplified reinforcement learning for a class of multiagent systems with unknown dynamics. IEEE Trans. Ind. Electron. 67(9), 7879–7888 (2020)

    Article  ADS  Google Scholar 

  7. Liu, D., Liu, H., Lü, J., Lewis, F.L.: Time-varying formation of heterogeneous multiagent systems via reinforcement learning subject to switching topologies. IEEE Trans. Circuits Syst. I Regul. Pap. 70(6), 2550–2560 (2023)

    Article  Google Scholar 

  8. Zhang, D., Wang, Y., Meng, L., Yan, J., Qin, C.: Adaptive critic design for safety-optimal ftc of unknown nonlinear systems with asymmetric constrained-input. ISA Trans. 155, 309–318 (2024)

    Article  Google Scholar 

  9. Qin, C., Ran, X., Zhang, D.: Unsupervised image stitching based on generative adversarial networks and feature frequency awareness algorithm. Appl. Soft Comput. 183, 113466 (2025)

    Article  Google Scholar 

  10. Qin, C., Qiao, X., Wang, J., Zhang, D., Hou, Y., Hu, S.: Barrier-critic adaptive robust control of nonzero-sum differential games for uncertain nonlinear systems with state constraints. IEEE Transactions on Systems, Man, and Cybernetics: Systems 54(1), 50–63 (2024)

    Article  ADS  Google Scholar 

  11. Qin, C., Jiang, K., Wang, Y., Zhu, T., Wu, Y., Zhang, D.: Event-triggered H-\(\infty \) control for unknown constrained nonlinear systems with application to robot arm. Appl. Math. Model. 144, 116089 (2025)

    Article  MathSciNet  Google Scholar 

  12. Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction, 2nd edn. MIT Press, Cambridge, MA (2018)

    Google Scholar 

  13. Peng, Z., Luo, R., Hu, J., Shi, K., Nguang, S.K., Ghosh, B.K.: Optimal tracking control of nonlinear multiagent systems using internal reinforce q-learning. IEEE Trans. Neural Netw. Learn. Syst. 33(8), 4043–4055 (2022)

    Article  MathSciNet  Google Scholar 

  14. Xu, W., Wang, L., Sun, S., Xia, C., Chen, Z.: A novel policy iteration algorithm for solving the optimal consensus control problem of a discrete-time multiagent system with unknown dynamics. Sci. China Inf. Sci. 66(8), 189204 (2023)

    Article  Google Scholar 

  15. Liu, Q., Yan, H., Wang, M., Li, Z., Liu, S.: Data-driven optimal bipartite consensus control for second-order multiagent systems via policy gradient reinforcement learning. IEEE Trans. Cybern. 54(6), 3468–3478 (2024)

    Article  Google Scholar 

  16. Zhang, C., Ji, L., Yang, S., Guo, X., Li, J., Li, H.: Time-varying formation optimization tracking of multi-agent systems with semi-markov switching topology. Nonlinear Dyn. 112(12), 10095–10108 (2024)

    Article  Google Scholar 

  17. Meng, H., Pang, D., Cao, J., Guo, Y., Niazi, A.U.K.: Optimal bipartite consensus control for heterogeneous unknown multi-agent systems via reinforcement learning. Appl. Math. Comput. 476, 128785 (2024)

    MathSciNet  Google Scholar 

  18. Lan, J., Liu, Y.-J., Yu, D., Wen, G., Tong, S., Liu, L.: Time-varying optimal formation control for second-order multiagent systems based on neural network observer and reinforcement learning. IEEE Trans. Neural Netw. Learn. Syst. 35(3), 3144–3155 (2024)

    Article  MathSciNet  Google Scholar 

  19. Jin, Y., Wang, F., Lai, G., Zhang, X.: Reinforcement learning-based adaptive optimal output feedback control for nonlinear systems with output quantization. Nonlinear Dyn. 113(7), 7029–7045 (2024)

    Article  Google Scholar 

  20. Si, W., Gao, S., Zhang, M., Wen, T., Bai, Y., Wang, H.: Approximate optimal control for uncertain nonlinear systems: A reinforcement relearning framework. Nonlinear Dyn., 1–28 (2025) https://doi.org/10.1007/s11071-025-11393-9

  21. Dimarogonas, D.V., Frazzoli, E., Johansson, K.H.: Distributed event-triggered control for multi-agent systems. IEEE Trans. Autom. Control 57(5), 1291–1297 (2012)

    Article  ADS  MathSciNet  Google Scholar 

  22. Xing, L., Wen, C., Liu, Z., Su, H., Cai, J.: Event-triggered adaptive control for a class of uncertain nonlinear systems. IEEE Trans. Autom. Control 62(4), 2071–2076 (2017)

    Article  ADS  MathSciNet  Google Scholar 

  23. Amirkhani, A., Barshooi, A.H.: Consensus in multi-agent systems: a review. Artif. Intell. Rev. 55(5), 3897–3935 (2021)

    Article  Google Scholar 

  24. Zhu, W., Pu, H., Wang, D., Li, H.: Event-based consensus of second-order multi-agent systems with discrete time. Automatica 79, 78–83 (2017)

    Article  MathSciNet  Google Scholar 

  25. Li, X., Bai, Y., Dong, X., Li, Q., Ren, Z.: Distributed time-varying formation control with uncertainties based on an event-triggered mechanism. Sci. China Inf. Sci. 64(3), 132204 (2021)

    Article  MathSciNet  Google Scholar 

  26. Tang, Y., Zhang, D., Shi, P., Zhang, W., Qian, F.: Event-based formation control for nonlinear multiagent systems under dos attacks. IEEE Trans. Autom. Control 66(1), 452–459 (2021)

    Article  ADS  MathSciNet  Google Scholar 

  27. Zhao, Y., Zhu, F., Xu, D.: Event-triggered bipartite time-varying formation control for multiagent systems with unknown inputs. IEEE Trans. Cybern. 53(9), 5904–5917 (2023)

    Article  Google Scholar 

  28. Zhu, W., Cao, W., Yan, M., Li, Q.: Event-triggered formation control of multiagent systems with linear continuous-time dynamic models. IEEE Trans. Syst. Man Cybern. Syst. 52(10), 6235–6245 (2022)

    Article  ADS  Google Scholar 

  29. Zhang, Y., Zhang, J., Xiong, J.: Event-triggered cooperative robust formation control of multi-agent systems via reinforcement learning. Appl. Intell. 54(17), 8367–8383 (2024)

    Article  Google Scholar 

  30. Liu, C., Liu, L.: Finite-horizon robust event-triggered control for nonlinear multi-agent systems with state delay. Neural Process. Lett. 55(4), 5167–5191 (2022)

    Article  ADS  Google Scholar 

  31. Liu, C., Liu, L., Cao, J., Abdel-Aty, M.: Intermittent event-triggered optimal leader-following consensus for nonlinear multi-agent systems via actor-critic algorithm. IEEE Trans. Neural Netw. Learn. Syst. 34(8), 3992–4006 (2023)

    Article  MathSciNet  Google Scholar 

  32. Dou, L., Cai, S., Zhang, X., Su, X., Zhang, R.: Event-triggered-based adaptive dynamic programming for distributed formation control of multi-uav. J. Franklin Inst. 359(8), 3671–3691 (2022)

    Article  ADS  MathSciNet  Google Scholar 

  33. Zhao, W., Yu, W., Zhang, H.: Event-triggered optimal consensus tracking control for multi-agent systems with unknown internal states and disturbances. Nonlin. Anal. Hybrid Syst. 33, 227–248 (2019)

    Article  MathSciNet  Google Scholar 

  34. Guo, X., Yan, W., Cui, R.: Event-triggered reinforcement learning-based adaptive tracking control for completely unknown continuous-time nonlinear systems. IEEE Trans. Cybern. 50(7), 3231–3242 (2020)

    Article  Google Scholar 

  35. Li, Y., Wang, X., Sun, J., Wang, G., Chen, J.: Data-driven consensus control of fully distributed event-triggered multi-agent systems. Sci. China Inf. Sci. 66(5), 152202 (2023)

    Article  MathSciNet  Google Scholar 

  36. Xu, Y., Sun, J., Pan, Y.-J., Wu, Z.-G.: Optimal tracking control of heterogeneous mass using event-driven adaptive observer and reinforcement learning. IEEE Trans. Neural Netw. Learn. Syst. 35(4), 5577–5587 (2024)

    Article  MathSciNet  Google Scholar 

  37. Peng, Z., Luo, R., Hu, J., Shi, K., Ghosh, B.K.: Distributed optimal tracking control of discrete-time multiagent systems via event-triggered reinforcement learning. IEEE Trans. Circuits Syst. I Regul. Pap. 69(9), 3689–3700 (2022)

    Article  Google Scholar 

  38. Ji, L., Li, X., Zhang, C., Xiao, Y., Li, H.: Optimal group consensus control for the second-order agents in the coopetition networks via adaptive dynamic programming and event-triggered methods. Optim. Control Appl. Methods 43(5), 1546–1567 (2022)

    Article  MathSciNet  Google Scholar 

  39. Zhang, C., Ji, L., Yang, S., Guo, X., Li, H.: Distributed optimal consensus control for multiagent systems based on event-triggered and prioritized experience replay strategies. Sci. China Inf. Sci. 68(1), 112206 (2024)

    Article  MathSciNet  Google Scholar 

  40. Ming, Z., Zhang, H., Yan, Y., Sun, J.: Self-triggered adaptive dynamic programming for model-free nonlinear systems via generalized fuzzy hyperbolic model. IEEE Trans. Syst. Man Cybern. Syst. 53(5), 2792–2801 (2023)

  41. Zhao, B., Zhang, S., Liu, D.: Self-triggered approximate optimal neuro-control for nonlinear systems through adaptive dynamic programming. IEEE Trans. Neural Netw. Learn. Syst. 36(3), 4713–4723 (2025)

    Article  MathSciNet  Google Scholar 

  42. Li, Y., Wang, X., Sun, J., Wang, G., Chen, J.: Self-triggered consensus control of multiagent systems from data. IEEE Trans. Autom. Control 69(7), 4702–4709 (2024)

  43. Zhang, H., Jiang, H., Luo, Y., Xiao, G.: Data-driven optimal consensus control for discrete-time multi-agent systems with unknown dynamics using reinforcement learning method. IEEE Trans. Ind. Electron. 64(5), 4091–4100 (2017)

    Article  ADS  Google Scholar 

  44. Lewis, F.L., Vrabie, D., Syrmos, V.L.: Optimal Control, 3rd edn. John Wiley & Sons, New York (2012)

    Book  Google Scholar 

  45. Abouheaf, M.I., Lewis, F.L., Vamvoudakis, K.G., Haesaert, S., Babuska, R.: Multi-agent discrete-time graphical games and reinforcement learning solutions. Automatica 50(12), 3038–3053 (2014)

    Article  MathSciNet  Google Scholar 

  46. Liu, D., Wei, Q.: Policy iteration adaptive dynamic programming algorithm for discrete-time nonlinear systems. IEEE Trans. Neural Netw. Learn. Syst. 25(3), 621–634 (2014)

    Article  Google Scholar 

  47. Hornik, K., Stinchcombe, M., White, H.: Universal approximation of an unknown mapping and its derivatives using multilayer feedforward networks. Neural Netw. 3(5), 551–560 (1990)

    Article  Google Scholar 

  48. Cybenko, G.: Approximation by superpositions of a sigmoidal function. Math. Control Signals Syst. 5(4), 455–455 (1992)

    Article  MathSciNet  Google Scholar 

  49. Zhao, S.: Mathematical Foundations of Reinforcement Learning. Springer, Beijing, China (2024)

    Google Scholar 

Download references

Funding

This work was jointly supported by National Natural Science Foundation of China (Grant No. 62373071) and Natural Science Foundation of Chongqing, China (Grant No. CSTB2023NSCQ-LZX0075).

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing, 400065, China

    Huizhu Pu

  2. Key Laboratory of Intelligent Analysis and Decision on Complex Systems, Chongqing University of Posts and Telecommunications, Chongqing, 400065, China

    Huizhu Pu & Wei Zhu

  3. College of Electronic and Information Engineering, Southwest University, Chongqing, 400715, China

    Run Tang

  4. School of Mathematics and Statistics, Shandong Normal University, Shandong, 276005, China

    Xiaodi Li

Authors
  1. Huizhu Pu
    View author publications

    Search author on:PubMed Google Scholar

  2. Wei Zhu
    View author publications

    Search author on:PubMed Google Scholar

  3. Run Tang
    View author publications

    Search author on:PubMed Google Scholar

  4. Xiaodi Li
    View author publications

    Search author on:PubMed Google Scholar

Contributions

Huizhu Pu contributed to the conceptualization, methodology, software, investigation, formal analysis, data curation, and writing the original draft & review. Wei Zhu was responsible for project administration, conceptualization, methodology, validation, formal analysis, writing review & editing, supervision and funding acquisition. Run Tang contributed to conceptualization, methodology, data curation, formal analysis. Xiaodi Li contributed to methodology, data curation, resources, validation.

Corresponding author

Correspondence to Wei Zhu.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Pu, H., Zhu, W., Tang, R. et al. Optimal time-varying formation tracking control for nonlinear multi-agent systems via event-triggered and self-triggered reinforcement learning. Nonlinear Dyn 114, 37 (2026). https://doi.org/10.1007/s11071-025-11935-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Version of record:

  • DOI: https://doi.org/10.1007/s11071-025-11935-1

Keywords