Data-Driven Hierarchical Contextual Control for Mobile Autonomous Systems
Autonomous systems, such as drones, cars, and robots, operate without human control and are increasingly used in various domains like smart homes, agriculture, and logistics. Traditional control theory aims to guide systems to follow desired references, but controlling autonomous systems, particularly Multi-Agent Systems (MAS), is more complex due to partial information, conflicting objectives, and the need for agents to decide between cooperation and competition. Model-free reinforcement learning (RL) has gained popularity for its ability to control these systems without prior knowledge of system dynamics, framing problems as Markov Decision Processes (MDPs). However, MDPs lack a hierarchical perspective, which is essential for capturing the inherent structure of autonomous systems and their environments, allowing for task transfer and efficient control. RL's current limitations include its focus on episodic tasks, while real-world applications demand systems that adapt to time-varying objectives and context changes. To address these challenges, we envisoned a novel framework of hierarchical contextual MDPs to model complex, dynamic scenarios involving single or multi-agent systems. This framework aims to enable adaptive, data-driven control solutions, offering a rich avenue for research in algorithm design, mathematical analysis, and practical implementation.
The framework of hierarchical contextual MDPs addresses diverse research problems aimed at transforming mobile systems into fully autonomous ones capable of solving tasks reliably and provably in response to user feedback or contextual changes. These systems, such as drones, cleaning robots, or personal assistants, combine mobility, perception, and multi-contextual task-solving capabilities. The proposed architecture includes a contextual learning agent for high-level decision-making, a reference governor to enforce constraints, and a low-level model-based controller for reliable execution. This design allows the RL agent to focus on effective, model-free learning and enables knowledge transfer between contexts, ensuring both adaptability and system reliability.
Fundamental long-term research questions that we seek to answere include:
- How to design an RL algorithm framework so agents can learn efficiently from different contexts?
- How to automatically create, divide, and detect contexts for autonomous mobile systems from data?
- How to incorporate low-level control information explicitely into a contextual learning process?
- How to train a contextual control architecture end-to-end to ensure provable stability and convergence?
This project is supported by the Commission for Research and Young Scientists and the Faculty of Computer Science, Electrical Engineering and Mathematics at Paderborn University.
Open SHK/WHB Position in the project
Last edited on 05.12.2024