March 26, 2021

[Guest Lecture @CMU] Deep Reinforcement Learning: Building Autonomous Agents

Delivered a quick primer on Deep Reinforcement Learning and any interested and curious student can dive into the field using read-made recipes from the TFRL Cookbook. The talk also covers pointers to free book, lectures and courses available to build a strong foundation and theoretical understanding before embarking on a practical, application oriented journey using the relatively advanced recipes discussed in the TFRL Cookbook.

March 9, 2021

[Patent]Spatial and temporal attention-based deep reinforcement learning of hierarchical lane-change policies for controlling an autonomous vehicle

Performing safe and efficient lane changes is a crucial feature for creating fully autonomous vehicles. Recent advances have demonstrated successful lane following behavior using deep reinforcement learning, yet the interactions with other vehicles on road for lane changes are rarely considered. Systems and methods are provided that employ spatial and temporal attention-based deep reinforcement learning of hierarchical lane-change policies for controlling an autonomous vehicle. An actor-critic network architecture includes an actor network that process image data received from an environment to learn the lane-change policies as a set of hierarchical actions, and a critic network that evaluates the lane-change policies to calculate loss and gradients to predict an action-value function (Q) that is used to drive learning and update parameters of the lane-change policies. The actor-critic network architecture implements a spatial attention module to select relevant regions in the image data that are of importance, and a temporal attention module to learn temporal attention weights to be applied to past frames of image data to indicate relative importance in deciding which lane-change policy to select.

November 6, 2020

[Talk]FIU Seminar on Multi-Agent Deep Reinforcement Learning for Connected Autonomous Driving

Delivered seminar at Florida Internation University School of Computing and Information Sciences (FIUSCIS). Abstract: The ability to autonomously navigate in 2D, 3D and unconstrained spaces by vehicles, robots or agents is desirable for several real-world applications. Autonomous driving on roads, which is a subset of the autonomous navigation space has become one of the major focus in the automotive industry in the recent times in addition to electrification. It involves autonomous vehicles navigating safely and socially from their start location to their desired goal location in usually complex environments. The autonomous driving field has advanced to the point of feasible deployments in the real-world. But they are limited in several ways including their domain of operation. The capability to learn and adapt to changes in the driving environment and in the intents of other road actors is crucial for autonomous driving systems to scale beyond the current, limited operation design domains. With the increasingly ubiquitous availability of 5G communication infrastructure, connectivity among vehicles provides a whole new avenue for connected autonomous driving. This talk is on using multi-agent deep reinforcement learning as a framework for formulating autonomous driving problems and developing solutions for these problems using simulation. This talk proposes the use of Partially Observable Markov Games for formulating the connected autonomous driving problems with realistic assumptions. The taxonomy of multi-agent learning environments based on the nature of tasks, nature of agents and the nature of the environment to help in categorizing various autonomous driving problems that can be addressed under the proposed formulation will be discussed. In addition, MACAD-Gym, a multi-agent learning platform with an extensible set of Connected Autonomous Driving (CAD) simulation environments that enable the research and development of Deep RL based integrated sensing, perception, planning and control algorithms for CAD systems with unlimited operational design domain under realistic, multi-agent settings will also be discussed. The talk concludes with remarks on autonomous navigation in 3D space, AirSim, Bonsai and an overview of Microsoft Autonomous Systems.

July 24, 2020

[IJCNN20][Oral]Multi-Agent Deep RL for Connected Autonomous Driving

Excited to be part of IEEE World Congress on Computational Intelligence (WCCI) 2020. I will making an oral presentation at the International Joint Conference on Convolutional Neural Networks (IJCNN) Special Session on Methods and Applications of Deep Reinforcement Learning to Autonomous Systems. More info and link to presentation: https://2020.wcci-virtual.org/presentation/oral/multi-agent-connected-autonomous-driving-using-deep-reinforcement-learning

January 11, 2020

[Demo] CES 2020

It was an invaluable CES 2020 experience as a part of Microsoft’s Autonomous Systems group collaborating with Bell Helicopter (https://lnkd.in/eGqndv9) in the showcase of Bell’s AerOS’s capabilities with a fully operational demo using scaled-down Nexus 4EX (https://lnkd.in/enJ5W5F) in a smart city. While it’s far from perfect, just seeing the TOL (Take-Off-Landing) and the NAV (Navigation) “Brains”, (providing an autonomous visual guidance system as a backup) deployed onto Bell’s drones that were demoed to the 150k+ CES audience was rewarding for me after months of hard-work on the AI pipeline. Thanks to Microsoft and Bell for providing the opportunity. This was my shortest development-to-deployment/production cycle of an autonomous (sub)system till date that was demoed publicly on the target hardware/robot!

October 4, 2019

Winner at TechCrunch Disrupt San Francisco 2019 Hackathon

Built SaveWise- An AI powered fin-tech app that won the TechCrunch Disrupt San Francisco 2019 Hackathanon sponsored by Plaid.SaveWise is an AI service that connects to a user’s bank/card accounts, trains on their past transaction data and predicts upcoming expenses and savings potential for the future. It is aimed at helping users to pay down their debt faster and attain financial freedom. It provides pro-active, actionable notifications, for e.g.: It can predict when and where you will head out to grab lunch and provides you more economical options without changing your lifestyle (same cuisine etc.). Visit the project page for more info: https://devpost.com/software/spendwise-69ik2c

June 16, 2019

[CVPR19]Attention-Based Hierarchical Deep Reinforcement Learning for Lane Change Behaviors in Autonomous Driving

Performing safe and efficient lane changes is a crucial feature for creating fully autonomous vehicles. Recent advances have demonstrated successful lane following behavior using deep reinforcement learning, yet the interactions with other vehicles on road for lane changes are rarely considered. In this paper, we design a hierarchical Deep Reinforcement Learning (DRL) algorithm to learn lane change behaviors in dense traffic. By breaking down overall behavior to sub-policies, faster and safer lane change actions can be learned. We also apply temporal and spatial attention to the DRL architecture, which helps the vehicle focus more on surrounding vehicles and leads to smoother lane change behavior. We conduct our experiments in the TORCS simulator and the results outperform the state-of-art deep reinforcement learning algorithm in various lane change scenarios. Citing: Chen, Y., Dong, C., Palanisamy, P., Mudalige, P., Muelling, K., & Dolan, J. M. (2019). Attention-Based Hierarchical Deep Reinforcement Learning for Lane Change Behaviors in Autonomous Driving. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (pp. 0-0).

November 4, 2018

[ITSC18]POMDP and Hierarchical Options MDP with Continuous Actions for Autonomous Driving at Intersections

When applying autonomous driving technology to real-world scenarios, environmental uncertainties make the development of decision-making algorithms difficult. Modeling the problem as a Partially Observable Markov Decision Process (POMDP) [1] allows the algorithm to consider these uncertainties in the decision process, which makes it more robust to real sensor characteristics. However, solving the POMDP with reinforcement learning (RL) [2] often requires storing a large number of observations. Furthermore, for continuous action spaces, the system is computationally inefficient. This paper addresses these problems by proposing to model the problem as an MDP and learn a policy with RL using hierarchical options (HOMDP). The suggested algorithm can store the state-action pairs and only uses current observations to solve a POMDP problem. We compare the results of to the time-to-collision method [3] and the proposed POMDP-with-LSTM method. Our results show that the HOMDP approach is able to improve the performance of the agent for a four-way intersection task with two-way stop signs. The HOMDP method can generate both higher-level discrete options and lower-level continuous actions with only the observations of the current step. Citing: Z. Qiao, K. Muelling, J. M. Dolan, P. Palanisamy and P. Mudalige, “POMDP and Hierarchical Options MDP with Continuous Actions for Autonomous Driving at Intersections,” 2018 IEEE 21st International Conference on Intelligent Transportation Systems, Maui, 2018

October 29, 2018

3rd rank – Full Stack Deep Learning certification exam

What was it?: Hands-on program for developers familiar with the basics of deep learning
When was it?: August 3 – 5, UC Berkeley, CA. I scored 119 out of 124 which was the 3rd highest score. The certification exam statistics:
Minimum: 23.0; Maximum: 124.0; Mean: 100.75; Median: 105.0; Standard Deviation: 20.54
The certification exam tested the following topics:Fundamentals of Deep Learning, Convnets, Sequences, Vision Applications, Troubleshooting Deep Learning implementations, Deep Learning Infrastructure, Deep Learning model deployment

June 26, 2018

[IV18]Learning Vehicle Surrounding-aware Lane-changing Behavior from Observed Trajectories

Predicting lane-changing intentions has long been a very active area of research in the autonomous driving community. However, most of the literature has focused on individual vehicles and did not consider both the neighbor information and the accumulated effects of vehicle history trajectories when making the predictions. We propose to apply a surrounding-aware LSTM algorithm for predicting the intention of a vehicle to perform a lane change that takes advantage of both vehicle past trajectories and their neighbor’s current states. We trained the model on real-world lane changing data and were able to show in simulation that these two components can lead not only to higher accuracy, but also to earlier lane-changing prediction time, which plays an important role in potentially improving the autonomous vehicle’s overall performance. keywords: {mobile robots;road safety;road vehicles;vehicle surrounding-aware lane-changing behavior;autonomous driving community;vehicle history trajectories;surrounding-aware LSTM algorithm;autonomous vehicle;lane-changing prediction time;lane-changing intentions;Trajectory;History;Prediction algorithms;Automobiles;Training;Predictive models;Feature extraction;LSTM;lane-change intention}, Citing: S. Su, K. Muelling, J. Dolan, P. Palanisamy and P. Mudalige, “Learning Vehicle Surrounding-aware Lane-changing Behavior from Observed Trajectories,” 2018 IEEE Intelligent Vehicles Symposium (IV), Changshu, 2018, pp. 1412-1417. doi: 10.1109/IVS.2018.8500445