March 26, 2021

[Guest Lecture @CMU] Deep Reinforcement Learning: Building Autonomous Agents

Delivered a quick primer on Deep Reinforcement Learning and any interested and curious student can dive into the field using read-made recipes from the TFRL Cookbook. The talk also covers pointers to free book, lectures and courses available to build a strong foundation and theoretical understanding before embarking on a practical, application oriented journey using the relatively advanced recipes discussed in the TFRL Cookbook.

March 9, 2021

[Patent]Spatial and temporal attention-based deep reinforcement learning of hierarchical lane-change policies for controlling an autonomous vehicle

Performing safe and efficient lane changes is a crucial feature for creating fully autonomous vehicles. Recent advances have demonstrated successful lane following behavior using deep reinforcement learning, yet the interactions with other vehicles on road for lane changes are rarely considered. Systems and methods are provided that employ spatial and temporal attention-based deep reinforcement learning of hierarchical lane-change policies for controlling an autonomous vehicle. An actor-critic network architecture includes an actor network that process image data received from an environment to learn the lane-change policies as a set of hierarchical actions, and a critic network that evaluates the lane-change policies to calculate loss and gradients to predict an action-value function (Q) that is used to drive learning and update parameters of the lane-change policies. The actor-critic network architecture implements a spatial attention module to select relevant regions in the image data that are of importance, and a temporal attention module to learn temporal attention weights to be applied to past frames of image data to indicate relative importance in deciding which lane-change policy to select.

November 6, 2020

[Talk]FIU Seminar on Multi-Agent Deep Reinforcement Learning for Connected Autonomous Driving

Delivered seminar at Florida Internation University School of Computing and Information Sciences (FIUSCIS). Abstract: The ability to autonomously navigate in 2D, 3D and unconstrained spaces by vehicles, robots or agents is desirable for several real-world applications. Autonomous driving on roads, which is a subset of the autonomous navigation space has become one of the major focus in the automotive industry in the recent times in addition to electrification. It involves autonomous vehicles navigating safely and socially from their start location to their desired goal location in usually complex environments. The autonomous driving field has advanced to the point of feasible deployments in the real-world. But they are limited in several ways including their domain of operation. The capability to learn and adapt to changes in the driving environment and in the intents of other road actors is crucial for autonomous driving systems to scale beyond the current, limited operation design domains. With the increasingly ubiquitous availability of 5G communication infrastructure, connectivity among vehicles provides a whole new avenue for connected autonomous driving. This talk is on using multi-agent deep reinforcement learning as a framework for formulating autonomous driving problems and developing solutions for these problems using simulation. This talk proposes the use of Partially Observable Markov Games for formulating the connected autonomous driving problems with realistic assumptions. The taxonomy of multi-agent learning environments based on the nature of tasks, nature of agents and the nature of the environment to help in categorizing various autonomous driving problems that can be addressed under the proposed formulation will be discussed. In addition, MACAD-Gym, a multi-agent learning platform with an extensible set of Connected Autonomous Driving (CAD) simulation environments that enable the research and development of Deep RL based integrated sensing, perception, planning and control algorithms for CAD systems with unlimited operational design domain under realistic, multi-agent settings will also be discussed. The talk concludes with remarks on autonomous navigation in 3D space, AirSim, Bonsai and an overview of Microsoft Autonomous Systems.

July 23, 2020

[Guest talk] Keynote presentation on RL for Smart Cities (AMSB2020)

Invited keynote speaker at the international virtual conference on AI & ML Applications in Smart Buildings (AMSB2020). Gave a talk on “Reinforcement Learning for Smart Cities”.

December 8, 2019

[NeurIPS19] Multi-Agent Connected Autonomous Driving using Deep Reinforcement Learning

The capability to learn and adapt to changes in the driving environment is crucial fordeveloping autonomous driving systems that are scalable beyond geo-fenced oper-ational design domains. Deep Reinforcement Learning (RL) provides a promisingand scalable framework for developing adaptive learning based solutions. Deep RLmethods usually model the problem as a (Partially Observable) Markov DecisionProcess in which an agent acts in a stationary environment to learn an optimalbehavior policy. However, driving involves complex interaction between multiple,intelligent (artificial or human) agents in a highly non-stationary environment. Inthis paper, we propose the use of Partially Observable Markov Games(POSG) forformulating the connected autonomous driving problems with realistic assumptions.We provide a taxonomy of multi-agent learning environments based on the natureof tasks, nature of agents and the nature of the environment to help in categorizingvarious autonomous driving problems that can be addressed under the proposedformulation. As our main contributions, we provide MACAD-Gym, a Multi-AgentConnected, Autonomous Driving agent learning platform for furthering research inthis direction. Our MACAD-Gym platform provides an extensible set of ConnectedAutonomous Driving (CAD) simulation environments that enable the research anddevelopment of Deep RL- based integrated sensing, perception, planning andcontrol algorithms for CAD systems with unlimited operational design domainunder realistic, multi-agent settings. We also share the MACAD-Agents that weretrained successfully using the MACAD-Gym platform to learn control policies formultiple vehicle agents in a partially observable, stop-sign controlled, 3-way urbanintersection environment with raw (camera) sensor observations. Paper: https://arxiv.org/abs/1911.04175 Code: https://github.com/praveen-palanisamy/macad-gym

June 17, 2019

Joined Microsoft as a Senior AI Engineer

Excited to join Microsoft’s Autonomous Systems, Business AI org as a Senior AI Engineer to work on developing end-to-end platform and services for real-world AI applications using Reinforcement Learning and Machine Teaching

January 8, 2019

[WACV19]Learning On-Road Visual Control for Self-Driving Vehicles with Auxiliary Tasks

A safe and robust on-road navigation system is a crucial component of achieving fully automated vehicles. NVIDIA recently proposed an End-to-End algorithm that can directly learn steering commands from raw pixels of a front camera by using one convolutional neural network. In this paper, we leverage auxiliary information aside from raw images and design a novel network structure, called Auxiliary Task Network (ATN), to help boost the driving performance while maintaining the advantage of minimal training data and an End-to-End training method. In this network, we introduce human prior knowledge into vehicle navigation by transferring features from image recognition tasks. Image semantic segmentation is applied as an auxiliary task for navigation. We consider temporal information by introducing an LSTM module and optical flow to the network. Finally, we combine vehicle kinematics with a sensor fusion step. We discuss the benefits of our method over state-of-the-art visual navigation methods both in the Udacity simulation environment and on the real-world Comma.ai dataset. Citing: Yilun Chen, Praveen Palanisamy, Priyantha Mudalige, Katharina Muelling: “Learning On-Road Visual Control for Self-Driving Vehicles with Auxiliary Tasks”, 2018; arXiv:1812.07760.

October 31, 2018

GM Award for invention – Vehicle state estimation failure detection using Deep Learning

GM award for protected (secret) method invention: “Apparatus and Methodology for Deep Learning Based State Estimation Failure Detection”, 2017

June 26, 2019

[IV18]Automatic Curriculum Generation for RL in Autonomous Vehicles in Urban Environment

We address the problem of learning autonomous driving behaviors in urban intersections using deep reinforcement learning (DRL). DRL has become a popular choice for creating autonomous agents due to its success in various tasks. However, as the problems tackled become more complex, the number of training iterations necessary increase drastically. Curriculum learning has been shown to reduce the required training time and improve the performance of the agent, but creating an optimal curriculum often requires human handcrafting. In this work, we learn a policy for urban intersection crossing using DRL and introduce a method to automatically generate the curriculum for the training process from a candidate set of tasks. We compare the performance of the automatically generated curriculum (AGC) training to those of randomly generated sequences and show that AGC can significantly reduce the training time while achieving similar or better performance. keywords: {learning (artificial intelligence);mobile robots;optimal curriculum;human handcrafting;urban intersection crossing;DRL;training process;automatically generated curriculum training;randomly generated sequences;autonomous vehicles;urban environment;autonomous driving behaviors;urban intersections;deep reinforcement learning;autonomous agents;training iterations necessary increase;curriculum learning;required training time;AGC;Training;Autonomous vehicles;Task analysis;Learning (artificial intelligence);Machine learning;Heuristic algorithms}, Citing: Z. Qiao, K. Muelling, J. M. Dolan, P. Palanisamy and P. Mudalige, “Automatically Generated Curriculum based Reinforcement Learning for Autonomous Vehicles in Urban Environment,” 2018 IEEE Intelligent Vehicles Symposium (IV), Changshu, 2018, pp. 1233-1238. doi: 10.1109/IVS.2018.8500603