AeroSim is a scalable, performant flight simulation platform designed for aerospace engineering and software development, seamlessly integrating with industry-standard tools through standardized interfaces and data formats. Its modular architecture supports AI workflows and autonomy development, enabling the generation of training data and scenario execution for testing. With support for the Functional Mockup Interface (FMI), AeroSim allows straightforward integration of custom dynamics models, sensor models, and controllers. The simulator employs a robust middleware layer (Kafka or DDS) for standardized data communication, coordinated by a high-performance orchestrator and synchronized simulation state data-store. This design facilitates interfacing with high-fidelity renderers like UE5, Omniverse’s RTX and modular component adjustments without system-wide modifications. AeroSim’s Python API ensures accessibility for rapid simulation setups while offering deep control for complex scenarios. AeroSim is a completely open-source platform, with permissive dual-licenses under Apache License 2.0 and MIT License for the source code and CC BY 4.0 License for the assets. AeroSim empowers users to advance autonomy in the aerial industry efficiently

Premier-TACO is a novel multitask feature representation learning methodology for enhancing few-shot policy learning in sequential decision-making tasks. It pretrains general feature representations using a subset of relevant multitask offline datasets, which can then be fine-tuned for specific tasks with minimal expert demonstrations. Building upon temporal action contrastive learning with an efficient negative sampling strategy, Premier-TACO achieves remarkable performance improvements: 101% over learn-from-scratch baselines and 24% over previous state-of-the-art pretraining methods on DeepMind Control Suite, and similar advantages on MetaWorld benchmarks. Link to Project page, Code and Paper

The New World Report Awards for the Software and Technology category for 2023 recognized me as an AI Engineering Specialist of the year 2023 (USA).

Google Deepmind’s Multi-agent reinforcement learning with matchmaking policies patent which was granted on 2023-04-11 cites 3 patents in total (including patent examiner’s citations) and one of them is my patent application US20200033868A1 which was granted in 2020-11-24. The fact that the invention by David Silver ( led AlphaGo, AlphaZero, etc.), Oriol Vinyals (led AlphaStar, AlphaCode) and Max Jaderberg cites my work, motivates me as my work is indeed impactful.

A paper on prompt-based, procedural generation of diverse datasets for training AI models for aerial autonomy applications. Published in Robotics and Autonomous Systems journal, Volume 166. Link to summary video:https://www.youtube.com/watch?v=eKpOh-K-NfQ. Link to paper.

Contributed to IEEE 3129 Standard as a member of the IEEE 3129 Standard for Robustness Testing and Evaluation of AI-based Service working group as part of the AI Standards Committee of the IEEE Computer Society, which developed the IEEE standard 3129. The standard is now approved and published. The standard is aimed at providing a framework for testing and evaluating the robustness of AI-based services.

Spoke at the Commercial Unmanned Aerial Vehicle (CUAV) news webinar on Transforming Infrastructure Inspection with Simulation and Autonomy. I was joined by John McKenna, Co-Founder & CEO of sees.ai and Timothy Reuter from Microsoft. Discussed a few key aspects on how running high-fidelity simulations at scale can accelerate mission planning, software & AI/ML model development and iteration cycles. Leveraging AI and the Autonomy Building blocks including pre-trained models that can be fine-tuned to build custom autonomy modules is a key enabler for accelerating the journey towards aerial autonomy. I also covered some of the key features and focus areas of the Microsoft Project AirSim platform that enables the entire end-to-end pipeline for aerial autonomy. I went over two specific application scenarios: 1. Cell Tower inspection and 2. Bridge inspection. Link to the Webinar page. Link to the recording of the webinar. A snapshot summary is available in the webinar handout slides. Post on LinkedIn.

Project AirSim is an End-to-End platform for aerial autonomy that’s powered by Simulation, planet-scale 3D assets and AI. My team & I announced Microsoft Project AirSim at the Farnborough International Airshow, 2022 in London.

Delivered a quick primer on Deep Reinforcement Learning and any interested and curious student can dive into the field using read-made recipes from the TFRL Cookbook. The talk also covers pointers to free book, lectures and courses available to build a strong foundation and theoretical understanding before embarking on a practical, application oriented journey using the relatively advanced recipes discussed in the TFRL Cookbook.

A practical guide with 50 short-&-sweet recipes to help you build, train and deploy learning agents for real-world applications. Full blog-post

Performing safe and efficient lane changes is a crucial feature for creating fully autonomous vehicles. Recent advances have demonstrated successful lane following behavior using deep reinforcement learning, yet the interactions with other vehicles on road for lane changes are rarely considered. Systems and methods are provided that employ spatial and temporal attention-based deep reinforcement learning of hierarchical lane-change policies for controlling an autonomous vehicle. An actor-critic network architecture includes an actor network that process image data received from an environment to learn the lane-change policies as a set of hierarchical actions, and a critic network that evaluates the lane-change policies to calculate loss and gradients to predict an action-value function (Q) that is used to drive learning and update parameters of the lane-change policies. The actor-critic network architecture implements a spatial attention module to select relevant regions in the image data that are of importance, and a temporal attention module to learn temporal attention weights to be applied to past frames of image data to indicate relative importance in deciding which lane-change policy to select.

Delivered seminar at Florida Internation University School of Computing and Information Sciences (FIUSCIS). Abstract: The ability to autonomously navigate in 2D, 3D and unconstrained spaces by vehicles, robots or agents is desirable for several real-world applications. Autonomous driving on roads, which is a subset of the autonomous navigation space has become one of the major focus in the automotive industry in the recent times in addition to electrification. It involves autonomous vehicles navigating safely and socially from their start location to their desired goal location in usually complex environments. The autonomous driving field has advanced to the point of feasible deployments in the real-world. But they are limited in several ways including their domain of operation. The capability to learn and adapt to changes in the driving environment and in the intents of other road actors is crucial for autonomous driving systems to scale beyond the current, limited operation design domains. With the increasingly ubiquitous availability of 5G communication infrastructure, connectivity among vehicles provides a whole new avenue for connected autonomous driving. This talk is on using multi-agent deep reinforcement learning as a framework for formulating autonomous driving problems and developing solutions for these problems using simulation. This talk proposes the use of Partially Observable Markov Games for formulating the connected autonomous driving problems with realistic assumptions. The taxonomy of multi-agent learning environments based on the nature of tasks, nature of agents and the nature of the environment to help in categorizing various autonomous driving problems that can be addressed under the proposed formulation will be discussed. In addition, MACAD-Gym, a multi-agent learning platform with an extensible set of Connected Autonomous Driving (CAD) simulation environments that enable the research and development of Deep RL based integrated sensing, perception, planning and control algorithms for CAD systems with unlimited operational design domain under realistic, multi-agent settings will also be discussed. The talk concludes with remarks on autonomous navigation in 3D space, AirSim, Bonsai and an overview of Microsoft Autonomous Systems.

Covered topics: Deep Reinforcement Learning (DRL or DeepRL) applied to the automotive industry, Simulation platforms and the role of simulators in training agents, Obtaining data to prepare the autonomous vehicle, Methods to evaluate robustness of the solution, Deploying in real world, Startups to use DL or be at the forefront of DL, Techcrunch Disrupt Hackathon win & engineers at hackathons as a practice

Excited to be part of IEEE World Congress on Computational Intelligence (WCCI) 2020. I will making an oral presentation at the International Joint Conference on Convolutional Neural Networks (IJCNN) Special Session on Methods and Applications of Deep Reinforcement Learning to Autonomous Systems. More info and link to presentation: https://2020.wcci-virtual.org/presentation/oral/multi-agent-connected-autonomous-driving-using-deep-reinforcement-learning

Invited keynote speaker at the international virtual conference on AI & ML Applications in Smart Buildings (AMSB2020). Gave a talk on “Reinforcement Learning for Smart Cities”.

It was an invaluable CES 2020 experience as a part of Microsoft’s Autonomous Systems group collaborating with Bell Helicopter (https://lnkd.in/eGqndv9) in the showcase of Bell’s AerOS’s capabilities with a fully operational demo using scaled-down Nexus 4EX (https://lnkd.in/enJ5W5F) in a smart city. While it’s far from perfect, just seeing the TOL (Take-Off-Landing) and the NAV (Navigation) “Brains”, (providing an autonomous visual guidance system as a backup) deployed onto Bell’s drones that were demoed to the 150k+ CES audience was rewarding for me after months of hard-work on the AI pipeline. Thanks to Microsoft and Bell for providing the opportunity. This was my shortest development-to-deployment/production cycle of an autonomous (sub)system till date that was demoed publicly on the target hardware/robot!

The capability to learn and adapt to changes in the driving environment is crucial fordeveloping autonomous driving systems that are scalable beyond geo-fenced oper-ational design domains. Deep Reinforcement Learning (RL) provides a promisingand scalable framework for developing adaptive learning based solutions. Deep RLmethods usually model the problem as a (Partially Observable) Markov DecisionProcess in which an agent acts in a stationary environment to learn an optimalbehavior policy. However, driving involves complex interaction between multiple,intelligent (artificial or human) agents in a highly non-stationary environment. Inthis paper, we propose the use of Partially Observable Markov Games(POSG) forformulating the connected autonomous driving problems with realistic assumptions.We provide a taxonomy of multi-agent learning environments based on the natureof tasks, nature of agents and the nature of the environment to help in categorizingvarious autonomous driving problems that can be addressed under the proposedformulation. As our main contributions, we provide MACAD-Gym, a Multi-AgentConnected, Autonomous Driving agent learning platform for furthering research inthis direction. Our MACAD-Gym platform provides an extensible set of ConnectedAutonomous Driving (CAD) simulation environments that enable the research anddevelopment of Deep RL- based integrated sensing, perception, planning andcontrol algorithms for CAD systems with unlimited operational design domainunder realistic, multi-agent settings. We also share the MACAD-Agents that weretrained successfully using the MACAD-Gym platform to learn control policies formultiple vehicle agents in a partially observable, stop-sign controlled, 3-way urbanintersection environment with raw (camera) sensor observations. Paper: https://arxiv.org/abs/1911.04175 Code: https://github.com/praveen-palanisamy/macad-gym

Built SaveWise- An AI powered fin-tech app that won the TechCrunch Disrupt San Francisco 2019 Hackathanon sponsored by Plaid.SaveWise is an AI service that connects to a user’s bank/card accounts, trains on their past transaction data and predicts upcoming expenses and savings potential for the future. It is aimed at helping users to pay down their debt faster and attain financial freedom. It provides pro-active, actionable notifications, for e.g.: It can predict when and where you will head out to grab lunch and provides you more economical options without changing your lifestyle (same cuisine etc.). Visit the project page for more info: https://devpost.com/software/spendwise-69ik2c

I’m happy to announce that my book, HOIAWOG! “Hands-On Intelligent Agents with OpenAI Gym: Your guide to developing AI agents using deep reinforcement learning”, made it to the Best Reinforcement Learning eBooks of All Time! compiled by BookAuthority. BookAuthority collects and ranks the best books in the world, and it is a great honor to get this kind of recognition. Thank you for all the reader’s support! You can learn more about the HOIAWOG book here. The source code for all the agents, algorithms and implementation details are available on GitHub. You can get a copy of the book from Amazon.

Excited to join Microsoft’s Autonomous Systems, Business AI org as a Senior AI Engineer to work on developing end-to-end platform and services for real-world AI applications using Reinforcement Learning and Machine Teaching

Performing safe and efficient lane changes is a crucial feature for creating fully autonomous vehicles. Recent advances have demonstrated successful lane following behavior using deep reinforcement learning, yet the interactions with other vehicles on road for lane changes are rarely considered. In this paper, we design a hierarchical Deep Reinforcement Learning (DRL) algorithm to learn lane change behaviors in dense traffic. By breaking down overall behavior to sub-policies, faster and safer lane change actions can be learned. We also apply temporal and spatial attention to the DRL architecture, which helps the vehicle focus more on surrounding vehicles and leads to smoother lane change behavior. We conduct our experiments in the TORCS simulator and the results outperform the state-of-art deep reinforcement learning algorithm in various lane change scenarios. Citing: Chen, Y., Dong, C., Palanisamy, P., Mudalige, P., Muelling, K., & Dolan, J. M. (2019). Attention-Based Hierarchical Deep Reinforcement Learning for Lane Change Behaviors in Autonomous Driving. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (pp. 0-0).

A safe and robust on-road navigation system is a crucial component of achieving fully automated vehicles. NVIDIA recently proposed an End-to-End algorithm that can directly learn steering commands from raw pixels of a front camera by using one convolutional neural network. In this paper, we leverage auxiliary information aside from raw images and design a novel network structure, called Auxiliary Task Network (ATN), to help boost the driving performance while maintaining the advantage of minimal training data and an End-to-End training method. In this network, we introduce human prior knowledge into vehicle navigation by transferring features from image recognition tasks. Image semantic segmentation is applied as an auxiliary task for navigation. We consider temporal information by introducing an LSTM module and optical flow to the network. Finally, we combine vehicle kinematics with a sensor fusion step. We discuss the benefits of our method over state-of-the-art visual navigation methods both in the Udacity simulation environment and on the real-world Comma.ai dataset. Citing: Yilun Chen, Praveen Palanisamy, Priyantha Mudalige, Katharina Muelling: “Learning On-Road Visual Control for Self-Driving Vehicles with Auxiliary Tasks”, 2018; arXiv:1812.07760.

When applying autonomous driving technology to real-world scenarios, environmental uncertainties make the development of decision-making algorithms difficult. Modeling the problem as a Partially Observable Markov Decision Process (POMDP) [1] allows the algorithm to consider these uncertainties in the decision process, which makes it more robust to real sensor characteristics. However, solving the POMDP with reinforcement learning (RL) [2] often requires storing a large number of observations. Furthermore, for continuous action spaces, the system is computationally inefficient. This paper addresses these problems by proposing to model the problem as an MDP and learn a policy with RL using hierarchical options (HOMDP). The suggested algorithm can store the state-action pairs and only uses current observations to solve a POMDP problem. We compare the results of to the time-to-collision method [3] and the proposed POMDP-with-LSTM method. Our results show that the HOMDP approach is able to improve the performance of the agent for a four-way intersection task with two-way stop signs. The HOMDP method can generate both higher-level discrete options and lower-level continuous actions with only the observations of the current step. Citing: Z. Qiao, K. Muelling, J. M. Dolan, P. Palanisamy and P. Mudalige, “POMDP and Hierarchical Options MDP with Continuous Actions for Autonomous Driving at Intersections,” 2018 IEEE 21st International Conference on Intelligent Transportation Systems, Maui, 2018

GM award for protected (secret) method invention: “Apparatus and Methodology for Deep Learning Based State Estimation Failure Detection”, 2017

What was it?: Hands-on program for developers familiar with the basics of deep learning
When was it?: August 3 – 5, UC Berkeley, CA. I scored 119 out of 124 which was the 3rd highest score. The certification exam statistics:
Minimum: 23.0; Maximum: 124.0; Mean: 100.75; Median: 105.0; Standard Deviation: 20.54
The certification exam tested the following topics:Fundamentals of Deep Learning, Convnets, Sequences, Vision Applications, Troubleshooting Deep Learning implementations, Deep Learning Infrastructure, Deep Learning model deployment

We address the problem of learning autonomous driving behaviors in urban intersections using deep reinforcement learning (DRL). DRL has become a popular choice for creating autonomous agents due to its success in various tasks. However, as the problems tackled become more complex, the number of training iterations necessary increase drastically. Curriculum learning has been shown to reduce the required training time and improve the performance of the agent, but creating an optimal curriculum often requires human handcrafting. In this work, we learn a policy for urban intersection crossing using DRL and introduce a method to automatically generate the curriculum for the training process from a candidate set of tasks. We compare the performance of the automatically generated curriculum (AGC) training to those of randomly generated sequences and show that AGC can significantly reduce the training time while achieving similar or better performance. keywords: {learning (artificial intelligence);mobile robots;optimal curriculum;human handcrafting;urban intersection crossing;DRL;training process;automatically generated curriculum training;randomly generated sequences;autonomous vehicles;urban environment;autonomous driving behaviors;urban intersections;deep reinforcement learning;autonomous agents;training iterations necessary increase;curriculum learning;required training time;AGC;Training;Autonomous vehicles;Task analysis;Learning (artificial intelligence);Machine learning;Heuristic algorithms}, Citing: Z. Qiao, K. Muelling, J. M. Dolan, P. Palanisamy and P. Mudalige, “Automatically Generated Curriculum based Reinforcement Learning for Autonomous Vehicles in Urban Environment,” 2018 IEEE Intelligent Vehicles Symposium (IV), Changshu, 2018, pp. 1233-1238. doi: 10.1109/IVS.2018.8500603

Predicting lane-changing intentions has long been a very active area of research in the autonomous driving community. However, most of the literature has focused on individual vehicles and did not consider both the neighbor information and the accumulated effects of vehicle history trajectories when making the predictions. We propose to apply a surrounding-aware LSTM algorithm for predicting the intention of a vehicle to perform a lane change that takes advantage of both vehicle past trajectories and their neighbor’s current states. We trained the model on real-world lane changing data and were able to show in simulation that these two components can lead not only to higher accuracy, but also to earlier lane-changing prediction time, which plays an important role in potentially improving the autonomous vehicle’s overall performance. keywords: {mobile robots;road safety;road vehicles;vehicle surrounding-aware lane-changing behavior;autonomous driving community;vehicle history trajectories;surrounding-aware LSTM algorithm;autonomous vehicle;lane-changing prediction time;lane-changing intentions;Trajectory;History;Prediction algorithms;Automobiles;Training;Predictive models;Feature extraction;LSTM;lane-change intention}, Citing: S. Su, K. Muelling, J. Dolan, P. Palanisamy and P. Mudalige, “Learning Vehicle Surrounding-aware Lane-changing Behavior from Observed Trajectories,” 2018 IEEE Intelligent Vehicles Symposium (IV), Changshu, 2018, pp. 1412-1417. doi: 10.1109/IVS.2018.8500445